2021-03-12T02:21:03.114 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-12T02:21:03.116 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --description rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} --verbose --archive /home/teuthworker/archive/yuriw-2021-03-09_20:35:33-rgw-wip-yuri4-testing-2021-03-09-1006-octopus-distro-basic-smithi/5951134 --name yuriw-2021-03-09_20:35:33-rgw-wip-yuri4-testing-2021-03-09-1006-octopus-distro-basic-smithi -- /home/teuthworker/archive/yuriw-2021-03-09_20:35:33-rgw-wip-yuri4-testing-2021-03-09-1006-octopus-distro-basic-smithi/5951134/orig.config.yaml 2021-03-12T02:21:03.153 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-03-12T02:21:03.206 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2021-03-09_20:35:33-rgw-wip-yuri4-testing-2021-03-09-1006-octopus-distro-basic-smithi/5951134 branch: wip-yuri4-testing-2021-03-09-1006-octopus description: rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: ceph-qa@ceph.io first_in_suite: false job_id: '5951134' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2021-03-09_20:35:33-rgw-wip-yuri4-testing-2021-03-09-1006-octopus-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: wip-yuri4-testing-2021-03-09-1006-octopus ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb rgw: compression type: random frontend: civetweb rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: wip-yuri4-testing-2021-03-09-1006-octopus sha1: 3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb owner: scheduled_yuriw@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb sleep_before_teardown: 0 suite: rgw suite_branch: wip-yuri4-testing-2021-03-09-1006-octopus suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb targets: smithi077.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgWjnm/iuIIjyr4cN1NZVVqXGxv1uNGNsn9pMzqAsfjgBCcQPK/LuPILtOZjDQo5bI2ch8/+/Y+iRPbmDlNKZegCoe/E9U0ccVreM3hIU1iM+A7NQf59jE++pSt0z/9KX1ZeD5ValQ79L1wDholUHsR5CKNq8t9uTah2Jv9LzrSAFhIHWD7gsiUtQV5Nh+zxiXElh2GoC6maYlFqBiJBM/SGU/hX8DwhFnXS7e4E4NshtegWeijAt93Zsw4Anpu1ChfqYwB4uJW4EsbZmcx2zpgNbisXclfgtf/udwb4qJQRE8XjO4wWaqCGtNh9s4wFZpSaVQGBvDLqvsaF85L4SN smithi114.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXjz7S6TEnBMVHdLOfCMhKgTnCP9W9q8QDBYiH63QhmXddO2kAhw1oD+J8nw92mrC6qbK66SSkhGRPGDRFzB7MiY1tw4fPWBuzrDC4WDnBFHuD2tOLcb9Lwoue6DlV+bzChQpW6RgRYscpZJpC99lSnLHxX3cI4vS3ScpDBrnGG4NfqFzdKE1Hs8TxOmV4PhqTZTEo08/V7/WKGS8zLfmKPm1/bPgbbl/cJ+xFfYc0ro8wd6aQmdK+eZJx7DYK13Lt27YHVLq/aeRBkuggXLYZHz4gQAwXwVf5NWq9sixagCZe5TzjdAdQjgPWXl63/W2SUSWJjzbMP5bYCGXqjqon tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master teuthology_sha1: 3082387bba74fcd24c9700593d10418152d53c97 timestamp: 2021-03-09_20:35:33 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2137 2021-03-12T02:21:03.206 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/qa; will attempt to use it 2021-03-12T02:21:03.208 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/qa/tasks 2021-03-12T02:21:03.209 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-03-12T02:21:03.210 INFO:teuthology.task.internal:Checking packages... 2021-03-12T02:21:03.232 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb' 2021-03-12T02:21:03.233 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-03-12T02:21:03.233 INFO:teuthology.packaging:ref: None 2021-03-12T02:21:03.233 INFO:teuthology.packaging:tag: None 2021-03-12T02:21:03.234 INFO:teuthology.packaging:branch: wip-yuri4-testing-2021-03-09-1006-octopus 2021-03-12T02:21:03.234 INFO:teuthology.packaging:sha1: 3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb 2021-03-12T02:21:03.235 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-yuri4-testing-2021-03-09-1006-octopus 2021-03-12T02:21:03.372 INFO:teuthology.task.internal:Found packages for ceph version 15.2.9-120-g3cd904d7-1bionic 2021-03-12T02:21:03.372 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-03-12T02:21:03.396 INFO:teuthology.task.internal:no buildpackages task found 2021-03-12T02:21:03.397 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-03-12T02:21:03.519 INFO:teuthology.task.internal:Saving configuration 2021-03-12T02:21:03.533 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-03-12T02:21:03.575 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-12T02:21:03.594 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-12 02:15:38.724920', 'description': '/home/teuthworker/archive/yuriw-2021-03-09_20:35:33-rgw-wip-yuri4-testing-2021-03-09-1006-octopus-distro-basic-smithi/5951134', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH1Vo3WYlMeUYFJfWTABLZeUEhul/+6u25cpXvayxSTT', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi077.front.sepia.ceph.com'} 2021-03-12T02:21:03.611 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-12 02:15:38.724941', 'description': '/home/teuthworker/archive/yuriw-2021-03-09_20:35:33-rgw-wip-yuri4-testing-2021-03-09-1006-octopus-distro-basic-smithi/5951134', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIME3Juw1gQzEB6zDYYasYqnN/8bGL9z50fkdjLlgarUZ', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi114.front.sepia.ceph.com'} 2021-03-12T02:21:03.611 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-03-12T02:21:03.627 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2021-03-12T02:21:03.627 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2021-03-12T02:21:03.628 INFO:teuthology.run_tasks:Running task console_log... 2021-03-12T02:21:03.749 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7e95d1ff28>, signals=[15]) 2021-03-12T02:21:03.751 INFO:teuthology.run_tasks:Running task internal.connect... 2021-03-12T02:21:03.757 INFO:teuthology.task.internal:Opening connections... 2021-03-12T02:21:03.757 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2021-03-12T02:21:03.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-12T02:21:03.878 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2021-03-12T02:21:03.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-12T02:21:03.951 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-03-12T02:21:03.959 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2021-03-12T02:21:04.065 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2021-03-12T02:21:04.065 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2021-03-12T02:21:04.115 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="Ubuntu" 2021-03-12T02:21:04.115 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-03-12T02:21:04.116 INFO:teuthology.orchestra.run.smithi077.stdout:ID=ubuntu 2021-03-12T02:21:04.116 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE=debian 2021-03-12T02:21:04.116 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-03-12T02:21:04.116 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="18.04" 2021-03-12T02:21:04.117 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-12T02:21:04.117 INFO:teuthology.orchestra.run.smithi077.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-12T02:21:04.117 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-12T02:21:04.117 INFO:teuthology.orchestra.run.smithi077.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-12T02:21:04.117 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_CODENAME=bionic 2021-03-12T02:21:04.118 INFO:teuthology.orchestra.run.smithi077.stdout:UBUNTU_CODENAME=bionic 2021-03-12T02:21:04.119 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2021-03-12T02:21:04.134 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2021-03-12T02:21:04.141 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2021-03-12T02:21:04.141 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2021-03-12T02:21:04.189 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="Ubuntu" 2021-03-12T02:21:04.189 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-03-12T02:21:04.189 INFO:teuthology.orchestra.run.smithi114.stdout:ID=ubuntu 2021-03-12T02:21:04.190 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE=debian 2021-03-12T02:21:04.190 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-03-12T02:21:04.190 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="18.04" 2021-03-12T02:21:04.190 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-12T02:21:04.190 INFO:teuthology.orchestra.run.smithi114.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-12T02:21:04.191 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-12T02:21:04.191 INFO:teuthology.orchestra.run.smithi114.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-12T02:21:04.191 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_CODENAME=bionic 2021-03-12T02:21:04.191 INFO:teuthology.orchestra.run.smithi114.stdout:UBUNTU_CODENAME=bionic 2021-03-12T02:21:04.192 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2021-03-12T02:21:04.207 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-03-12T02:21:04.218 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-03-12T02:21:04.270 INFO:teuthology.task.internal:Checking for old test directory... 2021-03-12T02:21:04.270 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2021-03-12T02:21:04.273 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2021-03-12T02:21:04.278 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-03-12T02:21:04.322 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-03-12T02:21:04.323 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2021-03-12T02:21:04.325 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2021-03-12T02:21:04.334 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-03-12T02:21:04.418 INFO:teuthology.run_tasks:Running task kernel... 2021-03-12T02:21:04.438 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2021-03-12T02:21:04.439 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-03-12T02:21:04.439 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2021-03-12T02:21:04.447 INFO:teuthology.orchestra.run.smithi077.stdout:4.15.0-136-generic 2021-03-12T02:21:04.447 INFO:teuthology.task.kernel:Running kernel on smithi077: 4.15.0-136-generic 2021-03-12T02:21:04.447 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-get clean 2021-03-12T02:21:04.648 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-get update 2021-03-12T02:21:04.919 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-12T02:21:04.919 INFO:teuthology.orchestra.run.smithi077.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-03-12T02:21:04.978 INFO:teuthology.orchestra.run.smithi077.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-03-12T02:21:05.248 INFO:teuthology.orchestra.run.smithi077.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-03-12T02:21:05.300 INFO:teuthology.orchestra.run.smithi077.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,573 kB] 2021-03-12T02:21:05.432 INFO:teuthology.orchestra.run.smithi077.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,228 kB] 2021-03-12T02:21:05.757 INFO:teuthology.orchestra.run.smithi077.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [926 kB] 2021-03-12T02:21:05.777 INFO:teuthology.orchestra.run.smithi077.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,912 kB] 2021-03-12T02:21:05.781 INFO:teuthology.orchestra.run.smithi077.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [302 kB] 2021-03-12T02:21:05.785 INFO:teuthology.orchestra.run.smithi077.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [245 kB] 2021-03-12T02:21:05.788 INFO:teuthology.orchestra.run.smithi077.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [17.7 kB] 2021-03-12T02:21:05.789 INFO:teuthology.orchestra.run.smithi077.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [31.7 kB] 2021-03-12T02:21:05.790 INFO:teuthology.orchestra.run.smithi077.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,113 kB] 2021-03-12T02:21:05.865 INFO:teuthology.orchestra.run.smithi077.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [979 kB] 2021-03-12T02:21:05.880 INFO:teuthology.orchestra.run.smithi077.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [24.3 kB] 2021-03-12T02:21:05.881 INFO:teuthology.orchestra.run.smithi077.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [266 kB] 2021-03-12T02:21:05.893 INFO:teuthology.orchestra.run.smithi077.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [35.3 kB] 2021-03-12T02:21:05.894 INFO:teuthology.orchestra.run.smithi077.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,561 kB] 2021-03-12T02:21:05.930 INFO:teuthology.orchestra.run.smithi077.stdout:Get:19 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [249 kB] 2021-03-12T02:21:05.933 INFO:teuthology.orchestra.run.smithi077.stdout:Get:20 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [19.1 kB] 2021-03-12T02:21:05.933 INFO:teuthology.orchestra.run.smithi077.stdout:Get:21 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,480 B] 2021-03-12T02:21:05.934 INFO:teuthology.orchestra.run.smithi077.stdout:Get:22 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [4,412 B] 2021-03-12T02:21:05.944 INFO:teuthology.orchestra.run.smithi077.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,719 kB] 2021-03-12T02:21:05.980 INFO:teuthology.orchestra.run.smithi077.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [11.6 kB] 2021-03-12T02:21:05.980 INFO:teuthology.orchestra.run.smithi077.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [24.9 kB] 2021-03-12T02:21:05.981 INFO:teuthology.orchestra.run.smithi077.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,464 B] 2021-03-12T02:21:07.413 INFO:teuthology.orchestra.run.smithi077.stdout:Fetched 12.5 MB in 2s (5,126 kB/s) 2021-03-12T02:21:08.878 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2021-03-12T02:21:08.924 DEBUG:teuthology.orchestra.run.smithi077:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-12T02:21:09.007 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2021-03-12T02:21:09.245 INFO:teuthology.orchestra.run.smithi077.stdout:Building dependency tree... 2021-03-12T02:21:09.246 INFO:teuthology.orchestra.run.smithi077.stdout:Reading state information... 2021-03-12T02:21:09.325 INFO:teuthology.orchestra.run.smithi077.stderr:E: Unable to locate package linux-image-current-generic 2021-03-12T02:21:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-12T02:21:09.330 DEBUG:teuthology.orchestra.run.smithi077:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-12T02:21:09.409 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2021-03-12T02:21:09.649 INFO:teuthology.orchestra.run.smithi077.stdout:Building dependency tree... 2021-03-12T02:21:09.650 INFO:teuthology.orchestra.run.smithi077.stdout:Reading state information... 2021-03-12T02:21:09.864 INFO:teuthology.orchestra.run.smithi077.stdout:linux-image-generic is already the newest version (4.15.0.136.123). 2021-03-12T02:21:09.864 INFO:teuthology.orchestra.run.smithi077.stdout:linux-image-generic set to manually installed. 2021-03-12T02:21:09.864 INFO:teuthology.orchestra.run.smithi077.stdout:0 upgraded, 0 newly installed, 0 to remove and 18 not upgraded. 2021-03-12T02:21:09.865 DEBUG:teuthology.orchestra.run.smithi077:> dpkg -s linux-image-generic 2021-03-12T02:21:09.883 INFO:teuthology.orchestra.run.smithi077.stdout:Package: linux-image-generic 2021-03-12T02:21:09.883 INFO:teuthology.orchestra.run.smithi077.stdout:Status: install ok installed 2021-03-12T02:21:09.884 INFO:teuthology.orchestra.run.smithi077.stdout:Priority: optional 2021-03-12T02:21:09.884 INFO:teuthology.orchestra.run.smithi077.stdout:Section: kernel 2021-03-12T02:21:09.884 INFO:teuthology.orchestra.run.smithi077.stdout:Installed-Size: 17 2021-03-12T02:21:09.884 INFO:teuthology.orchestra.run.smithi077.stdout:Maintainer: Ubuntu Kernel Team 2021-03-12T02:21:09.885 INFO:teuthology.orchestra.run.smithi077.stdout:Architecture: amd64 2021-03-12T02:21:09.885 INFO:teuthology.orchestra.run.smithi077.stdout:Source: linux-meta 2021-03-12T02:21:09.885 INFO:teuthology.orchestra.run.smithi077.stdout:Version: 4.15.0.136.123 2021-03-12T02:21:09.885 INFO:teuthology.orchestra.run.smithi077.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-03-12T02:21:09.885 INFO:teuthology.orchestra.run.smithi077.stdout:Depends: linux-image-4.15.0-136-generic, linux-modules-extra-4.15.0-136-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-12T02:21:09.886 INFO:teuthology.orchestra.run.smithi077.stdout:Recommends: thermald 2021-03-12T02:21:09.886 INFO:teuthology.orchestra.run.smithi077.stdout:Description: Generic Linux kernel image 2021-03-12T02:21:09.886 INFO:teuthology.orchestra.run.smithi077.stdout: This package will always depend on the latest generic kernel image 2021-03-12T02:21:09.887 INFO:teuthology.orchestra.run.smithi077.stdout: available. 2021-03-12T02:21:09.887 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-03-12T02:21:09.887 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2021-03-12T02:21:09.895 INFO:teuthology.orchestra.run.smithi114.stdout:4.15.0-136-generic 2021-03-12T02:21:09.895 INFO:teuthology.task.kernel:Running kernel on smithi114: 4.15.0-136-generic 2021-03-12T02:21:09.895 DEBUG:teuthology.orchestra.run.smithi114:> sudo apt-get clean 2021-03-12T02:21:09.987 DEBUG:teuthology.orchestra.run.smithi114:> sudo apt-get update 2021-03-12T02:21:10.246 INFO:teuthology.orchestra.run.smithi114.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-12T02:21:10.246 INFO:teuthology.orchestra.run.smithi114.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-03-12T02:21:10.310 INFO:teuthology.orchestra.run.smithi114.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-03-12T02:21:10.578 INFO:teuthology.orchestra.run.smithi114.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-03-12T02:21:10.681 INFO:teuthology.orchestra.run.smithi114.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [926 kB] 2021-03-12T02:21:10.819 INFO:teuthology.orchestra.run.smithi114.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,912 kB] 2021-03-12T02:21:11.048 INFO:teuthology.orchestra.run.smithi114.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,573 kB] 2021-03-12T02:21:11.161 INFO:teuthology.orchestra.run.smithi114.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [302 kB] 2021-03-12T02:21:11.179 INFO:teuthology.orchestra.run.smithi114.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [245 kB] 2021-03-12T02:21:11.186 INFO:teuthology.orchestra.run.smithi114.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [17.7 kB] 2021-03-12T02:21:11.187 INFO:teuthology.orchestra.run.smithi114.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [31.7 kB] 2021-03-12T02:21:11.188 INFO:teuthology.orchestra.run.smithi114.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,113 kB] 2021-03-12T02:21:11.224 INFO:teuthology.orchestra.run.smithi114.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [979 kB] 2021-03-12T02:21:11.225 INFO:teuthology.orchestra.run.smithi114.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,228 kB] 2021-03-12T02:21:11.251 INFO:teuthology.orchestra.run.smithi114.stdout:Get:15 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [249 kB] 2021-03-12T02:21:11.258 INFO:teuthology.orchestra.run.smithi114.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [19.1 kB] 2021-03-12T02:21:11.259 INFO:teuthology.orchestra.run.smithi114.stdout:Get:17 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,480 B] 2021-03-12T02:21:11.259 INFO:teuthology.orchestra.run.smithi114.stdout:Get:18 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [4,412 B] 2021-03-12T02:21:11.265 INFO:teuthology.orchestra.run.smithi114.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [24.3 kB] 2021-03-12T02:21:11.265 INFO:teuthology.orchestra.run.smithi114.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [266 kB] 2021-03-12T02:21:11.274 INFO:teuthology.orchestra.run.smithi114.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [35.3 kB] 2021-03-12T02:21:11.275 INFO:teuthology.orchestra.run.smithi114.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,719 kB] 2021-03-12T02:21:11.343 INFO:teuthology.orchestra.run.smithi114.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,561 kB] 2021-03-12T02:21:11.387 INFO:teuthology.orchestra.run.smithi114.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [24.9 kB] 2021-03-12T02:21:11.388 INFO:teuthology.orchestra.run.smithi114.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [11.6 kB] 2021-03-12T02:21:11.388 INFO:teuthology.orchestra.run.smithi114.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,464 B] 2021-03-12T02:21:12.846 INFO:teuthology.orchestra.run.smithi114.stdout:Fetched 12.5 MB in 2s (5,062 kB/s) 2021-03-12T02:21:14.273 INFO:teuthology.orchestra.run.smithi114.stdout:Reading package lists... 2021-03-12T02:21:14.319 DEBUG:teuthology.orchestra.run.smithi114:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-12T02:21:14.396 INFO:teuthology.orchestra.run.smithi114.stdout:Reading package lists... 2021-03-12T02:21:14.628 INFO:teuthology.orchestra.run.smithi114.stdout:Building dependency tree... 2021-03-12T02:21:14.629 INFO:teuthology.orchestra.run.smithi114.stdout:Reading state information... 2021-03-12T02:21:14.709 INFO:teuthology.orchestra.run.smithi114.stderr:E: Unable to locate package linux-image-current-generic 2021-03-12T02:21:14.710 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-12T02:21:14.711 DEBUG:teuthology.orchestra.run.smithi114:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-12T02:21:14.791 INFO:teuthology.orchestra.run.smithi114.stdout:Reading package lists... 2021-03-12T02:21:15.039 INFO:teuthology.orchestra.run.smithi114.stdout:Building dependency tree... 2021-03-12T02:21:15.039 INFO:teuthology.orchestra.run.smithi114.stdout:Reading state information... 2021-03-12T02:21:15.244 INFO:teuthology.orchestra.run.smithi114.stdout:linux-image-generic is already the newest version (4.15.0.136.123). 2021-03-12T02:21:15.244 INFO:teuthology.orchestra.run.smithi114.stdout:linux-image-generic set to manually installed. 2021-03-12T02:21:15.244 INFO:teuthology.orchestra.run.smithi114.stdout:0 upgraded, 0 newly installed, 0 to remove and 18 not upgraded. 2021-03-12T02:21:15.246 DEBUG:teuthology.orchestra.run.smithi114:> dpkg -s linux-image-generic 2021-03-12T02:21:15.268 INFO:teuthology.orchestra.run.smithi114.stdout:Package: linux-image-generic 2021-03-12T02:21:15.269 INFO:teuthology.orchestra.run.smithi114.stdout:Status: install ok installed 2021-03-12T02:21:15.269 INFO:teuthology.orchestra.run.smithi114.stdout:Priority: optional 2021-03-12T02:21:15.269 INFO:teuthology.orchestra.run.smithi114.stdout:Section: kernel 2021-03-12T02:21:15.269 INFO:teuthology.orchestra.run.smithi114.stdout:Installed-Size: 17 2021-03-12T02:21:15.270 INFO:teuthology.orchestra.run.smithi114.stdout:Maintainer: Ubuntu Kernel Team 2021-03-12T02:21:15.271 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: amd64 2021-03-12T02:21:15.271 INFO:teuthology.orchestra.run.smithi114.stdout:Source: linux-meta 2021-03-12T02:21:15.271 INFO:teuthology.orchestra.run.smithi114.stdout:Version: 4.15.0.136.123 2021-03-12T02:21:15.272 INFO:teuthology.orchestra.run.smithi114.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-03-12T02:21:15.272 INFO:teuthology.orchestra.run.smithi114.stdout:Depends: linux-image-4.15.0-136-generic, linux-modules-extra-4.15.0-136-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-12T02:21:15.272 INFO:teuthology.orchestra.run.smithi114.stdout:Recommends: thermald 2021-03-12T02:21:15.272 INFO:teuthology.orchestra.run.smithi114.stdout:Description: Generic Linux kernel image 2021-03-12T02:21:15.272 INFO:teuthology.orchestra.run.smithi114.stdout: This package will always depend on the latest generic kernel image 2021-03-12T02:21:15.273 INFO:teuthology.orchestra.run.smithi114.stdout: available. 2021-03-12T02:21:15.273 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-03-12T02:21:15.274 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2021-03-12T02:21:15.274 DEBUG:teuthology.orchestra.run.smithi077:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-12T02:21:15.298 INFO:teuthology.orchestra.run.smithi077.stdout:ttyS1 2021-03-12T02:21:15.322 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2021-03-12T02:21:15.323 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-12T02:21:15.341 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2021-03-12T02:21:15.361 INFO:teuthology.run_tasks:Running task internal.base... 2021-03-12T02:21:15.382 INFO:teuthology.task.internal:Creating test directory... 2021-03-12T02:21:15.383 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-12T02:21:15.385 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-12T02:21:15.409 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-03-12T02:21:15.605 INFO:teuthology.run_tasks:Running task internal.archive... 2021-03-12T02:21:15.647 INFO:teuthology.task.internal:Creating archive directory... 2021-03-12T02:21:15.648 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-12T02:21:15.651 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-12T02:21:15.692 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-03-12T02:21:15.704 INFO:teuthology.task.internal:Enabling coredump saving... 2021-03-12T02:21:15.704 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-12T02:21:15.734 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-12T02:21:15.753 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-12T02:21:15.763 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-12T02:21:15.763 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-12T02:21:15.764 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-12T02:21:15.766 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-03-12T02:21:15.775 INFO:teuthology.task.internal:Configuring sudo... 2021-03-12T02:21:15.776 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-12T02:21:15.815 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-12T02:21:15.830 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-03-12T02:21:15.843 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-03-12T02:21:15.844 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-12T02:21:15.874 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-12T02:21:15.880 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-12T02:21:15.926 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-12T02:21:15.975 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2021-03-12T02:21:15.976 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-12T02:21:16.066 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-12T02:21:16.074 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-12T02:21:16.122 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2021-03-12T02:21:16.122 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-12T02:21:16.187 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2021-03-12T02:21:16.189 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2021-03-12T02:21:16.269 INFO:teuthology.run_tasks:Running task internal.timer... 2021-03-12T02:21:16.277 INFO:teuthology.task.internal:Starting timer... 2021-03-12T02:21:16.278 INFO:teuthology.run_tasks:Running task pcp... 2021-03-12T02:21:16.319 INFO:teuthology.run_tasks:Running task selinux... 2021-03-12T02:21:16.364 DEBUG:teuthology.task.selinux:Excluding smithi077: OS 'ubuntu' does not support SELinux 2021-03-12T02:21:16.364 DEBUG:teuthology.task.selinux:Excluding smithi114: OS 'ubuntu' does not support SELinux 2021-03-12T02:21:16.365 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-03-12T02:21:16.365 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2021-03-12T02:21:16.365 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-03-12T02:21:16.366 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-03-12T02:21:16.413 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-03-12T02:21:16.414 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-03-12T02:21:16.433 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-03-12T02:21:16.439 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com,smithi114.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-03-12T02:21:16.815 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-03-12T02:21:17.668 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-03-12T02:21:19.727 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-03-12T02:21:19.750 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:19 +0000 (0:00:00.038) 0:00:00.038 ********** 2021-03-12T02:21:19.761 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:19 +0000 (0:00:00.011) 0:00:00.049 ********** 2021-03-12T02:21:21.551 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-03-12T02:21:21.553 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi114.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi114.front.sepia.ceph.com] 2021-03-12T02:21:21.624 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:21 +0000 (0:00:01.863) 0:00:01.913 ********** 2021-03-12T02:21:21.643 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi077.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. 2021-03-12T02:21:21.644 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] 2021-03-12T02:21:21.707 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:21 +0000 (0:00:00.082) 0:00:01.995 ********** 2021-03-12T02:21:21.906 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-12T02:21:21.926 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:21 +0000 (0:00:00.218) 0:00:02.214 ********** 2021-03-12T02:21:21.937 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-12T02:21:21.965 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:21 +0000 (0:00:00.038) 0:00:02.253 ********** 2021-03-12T02:21:22.587 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-03-12T02:21:22.588 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-03-12T02:21:22.611 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:22 +0000 (0:00:00.646) 0:00:02.900 ********** 2021-03-12T02:21:22.619 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-03-12T02:21:22.644 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:22 +0000 (0:00:00.032) 0:00:02.932 ********** 2021-03-12T02:21:23.004 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-03-12T02:21:23.006 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.072410", "end": "2021-03-12 02:21:22.846221", "rc": 0, "start": "2021-03-12 02:21:22.773811", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-03-12T02:21:23.028 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:23 +0000 (0:00:00.384) 0:00:03.316 ********** 2021-03-12T02:21:23.036 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.068960", "end": "2021-03-12 02:21:23.458508", "rc": 0, "start": "2021-03-12 02:21:23.389548", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-03-12T02:21:23.064 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:23 +0000 (0:00:00.035) 0:00:03.352 ********** 2021-03-12T02:21:23.334 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-03-12T02:21:23.336 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-12T02:21:23.358 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:23 +0000 (0:00:00.293) 0:00:03.646 ********** 2021-03-12T02:21:23.366 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-12T02:21:23.392 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:23 +0000 (0:00:00.034) 0:00:03.680 ********** 2021-03-12T02:21:24.314 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615515683.3985667-10005-137749935130070/source", "state": "file", "uid": 0} 2021-03-12T02:21:24.336 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:24 +0000 (0:00:00.944) 0:00:04.624 ********** 2021-03-12T02:21:24.346 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615515683.4355118-10011-280303681104208/source", "state": "file", "uid": 0} 2021-03-12T02:21:24.371 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:24 +0000 (0:00:00.034) 0:00:04.659 ********** 2021-03-12T02:21:24.820 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-03-12T02:21:24.822 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-03-12T02:21:24.823 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail 2021-03-12T02:21:24.825 INFO:teuthology.task.ansible.out:.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@ 2021-03-12T02:21:24.827 INFO:teuthology.task.ansible.out:aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-12T02:21:24.915 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-03-12T02:21:24.919 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-03-12T02:21:25.079 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:25 +0000 (0:00:00.708) 0:00:05.367 ********** 2021-03-12T02:21:25.108 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:25 +0000 (0:00:00.028) 0:00:05.396 ********** 2021-03-12T02:21:25.832 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi077.front.sepia.ceph.com] 2021-03-12T02:21:25.868 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] 2021-03-12T02:21:25.901 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-03-12T02:21:25.903 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:25 +0000 (0:00:00.794) 0:00:06.190 ********** 2021-03-12T02:21:25.968 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:25 +0000 (0:00:00.065) 0:00:06.256 ********** 2021-03-12T02:21:26.330 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-03-12T02:21:26.332 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.164572", "end": "2021-03-12 02:21:26.299638", "rc": 0, "start": "2021-03-12 02:21:26.135066", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} 2021-03-12T02:21:26.355 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:26 +0000 (0:00:00.386) 0:00:06.643 ********** 2021-03-12T02:21:26.362 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.139391", "end": "2021-03-12 02:21:26.323434", "rc": 0, "start": "2021-03-12 02:21:26.184043", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} 2021-03-12T02:21:26.387 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:26 +0000 (0:00:00.032) 0:00:06.675 ********** 2021-03-12T02:21:26.508 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-03-12T02:21:26.510 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.014155", "end": "2021-03-12 02:21:26.970910", "failed_when_result": false, "rc": 0, "start": "2021-03-12 02:21:26.956755", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-12T02:21:26.549 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:26 +0000 (0:00:00.161) 0:00:06.836 ********** 2021-03-12T02:21:26.558 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.017409", "end": "2021-03-12 02:21:26.392246", "failed_when_result": false, "rc": 0, "start": "2021-03-12 02:21:26.374837", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-12T02:21:26.585 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:26 +0000 (0:00:00.036) 0:00:06.873 ********** 2021-03-12T02:21:26.633 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-03-12T02:21:26.635 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:21:26.658 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:26 +0000 (0:00:00.073) 0:00:06.947 ********** 2021-03-12T02:21:26.688 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:21:26.722 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:26 +0000 (0:00:00.063) 0:00:07.010 ********** 2021-03-12T02:21:26.732 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-03-12T02:21:26.734 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:21:26.760 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:26 +0000 (0:00:00.038) 0:00:07.049 ********** 2021-03-12T02:21:26.801 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:21:26.832 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:26 +0000 (0:00:00.071) 0:00:07.120 ********** 2021-03-12T02:21:26.839 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-03-12T02:21:26.841 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:21:26.864 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:26 +0000 (0:00:00.032) 0:00:07.153 ********** 2021-03-12T02:21:26.915 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:21:26.941 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:26 +0000 (0:00:00.076) 0:00:07.229 ********** 2021-03-12T02:21:26.947 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-03-12T02:21:26.949 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:21:26.976 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:26 +0000 (0:00:00.034) 0:00:07.264 ********** 2021-03-12T02:21:27.015 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:21:27.041 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:27 +0000 (0:00:00.064) 0:00:07.329 ********** 2021-03-12T02:21:27.047 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-03-12T02:21:27.049 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:21:27.078 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:27 +0000 (0:00:00.032) 0:00:07.362 ********** 2021-03-12T02:21:27.122 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:21:27.146 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:27 +0000 (0:00:00.072) 0:00:07.435 ********** 2021-03-12T02:21:27.282 INFO:teuthology.task.ansible.out: 2021-03-12T02:21:27.283 INFO:teuthology.task.ansible.out:TASK [users : Set UID_MIN to 1001] ********************************************* 2021-03-12T02:21:27.285 INFO:teuthology.task.ansible.out: ok: [smithi077.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-12T02:21:27.318 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:27 +0000 (0:00:00.171) 0:00:07.606 ********** 2021-03-12T02:21:27.320 INFO:teuthology.task.ansible.out: 2021-03-12T02:21:27.351 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-12T02:21:27.376 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:27 +0000 (0:00:00.058) 0:00:07.664 ********** 2021-03-12T02:21:27.571 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-03-12T02:21:27.624 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-03-12T02:21:27.723 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-12T02:21:27.787 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-12T02:21:27.872 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-12T02:21:27.935 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-12T02:21:28.010 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-12T02:21:28.084 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-12T02:21:28.169 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-12T02:21:28.232 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-12T02:21:28.233 INFO:teuthology.task.ansible.out: 2021-03-12T02:21:28.323 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-12T02:21:28.376 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-12T02:21:28.461 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-12T02:21:28.524 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-12T02:21:28.608 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-12T02:21:28.683 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-12T02:21:28.757 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-03-12T02:21:28.831 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-03-12T02:21:28.832 INFO:teuthology.task.ansible.out: 2021-03-12T02:21:28.905 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-12T02:21:28.978 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-12T02:21:29.052 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-12T02:21:29.125 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-12T02:21:29.192 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-12T02:21:29.278 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-12T02:21:29.342 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-12T02:21:29.416 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-12T02:21:29.503 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-12T02:21:29.566 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-12T02:21:29.640 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-12T02:21:29.716 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-12T02:21:29.789 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-12T02:21:29.864 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-12T02:21:29.938 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-12T02:21:30.013 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-12T02:21:30.087 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-12T02:21:30.171 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-12T02:21:30.234 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-12T02:21:30.298 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-12T02:21:30.382 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-12T02:21:30.456 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-12T02:21:30.541 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-12T02:21:30.606 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-12T02:21:30.679 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-03-12T02:21:30.743 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-03-12T02:21:30.827 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-12T02:21:30.891 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-12T02:21:30.975 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-12T02:21:31.039 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-12T02:21:31.125 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-12T02:21:31.189 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-12T02:21:31.275 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-12T02:21:31.338 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-12T02:21:31.414 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-12T02:21:31.479 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-12T02:21:31.552 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-12T02:21:31.626 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-12T02:21:31.710 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-12T02:21:31.784 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-12T02:21:31.858 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-12T02:21:31.912 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-12T02:21:31.914 INFO:teuthology.task.ansible.out: 2021-03-12T02:21:31.998 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-12T02:21:32.052 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-12T02:21:32.163 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-12T02:21:32.208 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-12T02:21:32.320 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-12T02:21:32.379 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-12T02:21:32.455 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-12T02:21:32.530 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-12T02:21:32.624 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-12T02:21:32.671 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-12T02:21:32.791 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-12T02:21:32.840 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-12T02:21:32.966 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-03-12T02:21:33.011 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-03-12T02:21:33.128 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-12T02:21:33.156 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-12T02:21:33.272 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-12T02:21:33.308 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-12T02:21:33.420 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-12T02:21:33.453 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-12T02:21:33.562 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-12T02:21:33.594 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-12T02:21:33.724 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-12T02:21:33.758 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-12T02:21:33.876 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-12T02:21:33.909 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-12T02:21:34.014 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-12T02:21:34.047 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-12T02:21:34.162 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-12T02:21:34.195 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-12T02:21:34.303 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-12T02:21:34.336 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-12T02:21:34.452 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-12T02:21:34.484 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-12T02:21:34.632 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-12T02:21:34.632 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-12T02:21:34.780 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-03-12T02:21:34.782 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-03-12T02:21:34.930 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-12T02:21:34.931 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-12T02:21:35.080 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-12T02:21:35.102 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-12T02:21:35.219 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-12T02:21:35.253 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-12T02:21:35.368 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-12T02:21:35.392 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-12T02:21:35.537 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-12T02:21:35.538 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-12T02:21:35.675 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-03-12T02:21:36.059 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups 2021-03-12T02:21:36.061 INFO:teuthology.task.ansible.out:": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-03-12T02:21:36.132 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-12T02:21:36.154 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-12T02:21:36.271 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-12T02:21:36.303 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-12T02:21:36.430 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-12T02:21:36.452 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-12T02:21:36.578 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-12T02:21:36.600 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-12T02:21:36.741 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-12T02:21:36.753 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-12T02:21:36.891 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-12T02:21:36.903 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-12T02:21:37.041 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-12T02:21:37.043 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-12T02:21:37.189 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-12T02:21:37.196 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-12T02:21:37.339 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-12T02:21:37.351 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-12T02:21:37.490 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-12T02:21:37.503 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-12T02:21:37.641 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-12T02:21:37.643 INFO:teuthology.task.ansible.out: 2021-03-12T02:21:37.645 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-12T02:21:37.795 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-12T02:21:37.796 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-12T02:21:37.943 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-12T02:21:37.945 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-12T02:21:38.101 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-12T02:21:38.103 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-12T02:21:38.243 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-12T02:21:38.255 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-12T02:21:38.383 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-12T02:21:38.405 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-12T02:21:38.526 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-12T02:21:38.548 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-12T02:21:38.674 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-12T02:21:38.697 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-12T02:21:38.822 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-12T02:21:38.845 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-12T02:21:38.960 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-12T02:21:38.993 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-12T02:21:39.108 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-12T02:21:39.131 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-12T02:21:39.257 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-12T02:21:39.279 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-12T02:21:39.415 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-12T02:21:39.427 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-12T02:21:39.553 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-12T02:21:39.564 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-12T02:21:39.701 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-12T02:21:39.712 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-12T02:21:39.838 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-12T02:21:39.861 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-12T02:21:39.986 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-12T02:21:40.008 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-12T02:21:40.136 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-12T02:21:40.147 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-12T02:21:40.148 INFO:teuthology.task.ansible.out: 2021-03-12T02:21:40.286 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-12T02:21:40.288 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-12T02:21:40.434 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-12T02:21:40.434 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-12T02:21:40.596 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-12T02:21:40.749 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-03-12T02:21:40.762 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-12T02:21:40.899 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-12T02:21:40.911 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-03-12T02:21:41.038 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-12T02:21:41.071 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-12T02:21:41.190 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-12T02:21:41.213 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-12T02:21:41.339 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-12T02:21:41.350 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-12T02:21:41.478 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-12T02:21:41.501 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-12T02:21:41.618 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-12T02:21:41.662 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-12T02:21:41.767 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-12T02:21:41.800 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-12T02:21:41.917 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-12T02:21:41.953 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-12T02:21:42.065 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-12T02:21:42.120 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-12T02:21:42.217 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-12T02:21:42.303 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-12T02:21:42.367 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-12T02:21:42.451 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-12T02:21:42.525 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-12T02:21:42.603 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-12T02:21:42.667 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-12T02:21:42.751 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-12T02:21:42.826 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-12T02:21:42.900 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-12T02:21:42.974 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-12T02:21:43.060 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-12T02:21:43.114 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-12T02:21:43.239 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-12T02:21:43.272 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-12T02:21:43.398 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-12T02:21:43.420 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-12T02:21:43.551 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-12T02:21:43.572 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-12T02:21:43.691 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-12T02:21:43.723 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-12T02:21:43.839 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-12T02:21:43.861 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-12T02:21:43.987 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-12T02:21:44.020 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-12T02:21:44.135 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-12T02:21:44.167 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-12T02:21:44.283 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-12T02:21:44.305 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-12T02:21:44.422 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-12T02:21:44.444 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-12T02:21:44.585 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-12T02:21:44.596 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-12T02:21:44.716 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-12T02:21:44.738 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-12T02:21:44.864 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-12T02:21:44.886 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-12T02:21:45.012 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-12T02:21:45.034 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-12T02:21:45.160 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-12T02:21:45.171 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-12T02:21:45.307 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-12T02:21:45.319 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-12T02:21:45.458 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-12T02:21:45.470 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-12T02:21:45.607 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-12T02:21:45.753 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-12T02:21:45.755 INFO:teuthology.task.ansible.out: ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-12T02:21:45.903 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-03-12T02:21:45.904 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-12T02:21:46.044 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-12T02:21:46.056 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-12T02:21:46.207 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-12T02:21:46.222 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-12T02:21:46.356 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-12T02:21:46.379 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-12T02:21:46.497 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-12T02:21:46.519 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-12T02:21:46.669 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-12T02:21:46.703 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-12T02:21:46.881 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-12T02:21:47.017 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-12T02:21:47.039 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-12T02:21:47.209 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-12T02:21:47.221 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-12T02:21:47.383 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-12T02:21:47.384 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-12T02:21:47.542 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-12T02:21:47.573 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-12T02:21:47.702 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-12T02:21:47.771 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-12T02:21:47.942 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-12T02:21:47.944 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-12T02:21:48.080 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-12T02:21:48.166 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-03-12T02:21:48.223 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-03-12T02:21:48.307 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-12T02:21:48.371 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-12T02:21:48.455 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-03-12T02:21:48.519 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-03-12T02:21:48.604 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-03-12T02:21:48.839 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-03-12T02:21:48.944 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:48 +0000 (0:00:21.567) 0:00:29.232 ********** 2021-03-12T02:21:48.967 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:48 +0000 (0:00:00.022) 0:00:29.255 ********** 2021-03-12T02:21:49.010 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-03-12T02:21:49.035 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:49 +0000 (0:00:00.068) 0:00:29.323 ********** 2021-03-12T02:21:49.069 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:49 +0000 (0:00:00.034) 0:00:29.358 ********** 2021-03-12T02:21:49.119 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-03-12T02:21:49.121 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-12T02:21:49.124 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-12T02:21:49.125 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-12T02:21:49.127 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-12T02:21:49.128 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-12T02:21:49.130 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-12T02:21:49.132 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a 2021-03-12T02:21:49.132 INFO:teuthology.task.ansible.out:4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e 2021-03-12T02:21:49.134 INFO:teuthology.task.ansible.out:7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef14 2021-03-12T02:21:49.136 INFO:teuthology.task.ansible.out:80c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d301 2021-03-12T02:21:49.137 INFO:teuthology.task.ansible.out:7b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214 2021-03-12T02:21:49.139 INFO:teuthology.task.ansible.out:bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl 2021-03-12T02:21:49.140 INFO:teuthology.task.ansible.out:@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelope 2021-03-12T02:21:49.141 INFO:teuthology.task.ansible.out:z@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed 2021-03-12T02:21:49.143 INFO:teuthology.task.ansible.out:19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-12T02:21:49.170 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:49 +0000 (0:00:00.100) 0:00:29.458 ********** 2021-03-12T02:21:49.181 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-12T02:21:49.183 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-12T02:21:49.185 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-12T02:21:49.187 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-12T02:21:49.188 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-12T02:21:49.190 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-12T02:21:49.191 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e 2021-03-12T02:21:49.193 INFO:teuthology.task.ansible.out:7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef14 2021-03-12T02:21:49.195 INFO:teuthology.task.ansible.out:80c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d301 2021-03-12T02:21:49.196 INFO:teuthology.task.ansible.out:7b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214 2021-03-12T02:21:49.198 INFO:teuthology.task.ansible.out:bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelope 2021-03-12T02:21:49.200 INFO:teuthology.task.ansible.out:z@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed 2021-03-12T02:21:49.202 INFO:teuthology.task.ansible.out:19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174f 2021-03-12T02:21:49.203 INFO:teuthology.task.ansible.out:e16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-12T02:21:49.233 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:49 +0000 (0:00:00.063) 0:00:29.522 ********** 2021-03-12T02:21:50.412 INFO:teuthology.task.ansible.out:FAILED - RETRYING: Clone the keys repo (5 retries left). 2021-03-12T02:21:50.434 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-03-12T02:21:50.435 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 1, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-12T02:21:50.458 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:21:50 +0000 (0:00:01.224) 0:00:30.746 ********** 2021-03-12T02:21:50.704 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:21:50.706 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-12T02:21:50.839 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-12T02:21:50.983 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-12T02:21:51.154 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-03-12T02:21:51.156 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-03-12T02:21:51.291 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-12T02:21:51.432 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-03-12T02:21:51.434 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-03-12T02:21:51.572 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-03-12T02:21:51.718 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-03-12T02:21:51.876 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-03-12T02:21:51.878 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:21:52.019 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-12T02:21:52.020 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-03-12T02:21:52.156 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-12T02:21:52.158 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-03-12T02:21:52.167 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:21:52.178 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:21:52.313 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-03-12T02:21:52.315 INFO:teuthology.task.ansible.out: 2021-03-12T02:21:52.472 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-12T02:21:52.474 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-12T02:21:52.611 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-12T02:21:52.740 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-12T02:21:52.742 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-03-12T02:21:52.923 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-03-12T02:21:53.124 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-12T02:21:53.304 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-12T02:21:53.445 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-03-12T02:21:53.447 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-03-12T02:21:53.595 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-12T02:21:53.596 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-03-12T02:21:53.722 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-12T02:21:53.862 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-12T02:21:54.003 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-12T02:21:54.005 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-12T02:21:54.168 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-12T02:21:54.320 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-12T02:21:54.456 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-12T02:21:54.594 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-12T02:21:54.749 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-12T02:21:54.904 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-12T02:21:54.906 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-03-12T02:21:55.054 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-03-12T02:21:55.203 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-03-12T02:21:55.350 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-03-12T02:21:55.540 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-03-12T02:21:55.735 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-12T02:21:55.893 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-12T02:21:56.067 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-12T02:21:56.205 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-12T02:21:56.381 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-12T02:21:56.557 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-12T02:21:56.559 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-12T02:21:56.692 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-12T02:21:56.694 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-12T02:21:56.840 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-12T02:21:57.019 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-12T02:21:57.145 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-12T02:21:57.284 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-12T02:21:57.285 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-12T02:21:57.444 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-12T02:21:57.446 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-12T02:21:57.447 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-12T02:21:57.622 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-12T02:21:57.791 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-12T02:21:57.939 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-12T02:21:58.076 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-12T02:21:58.217 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-12T02:21:58.358 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-12T02:21:58.360 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-03-12T02:21:58.485 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-12T02:21:58.632 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-12T02:21:58.634 INFO:teuthology.task.ansible.out: 2021-03-12T02:21:58.770 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-12T02:21:58.772 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-03-12T02:21:58.909 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-12T02:21:59.061 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-12T02:21:59.200 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-12T02:21:59.203 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-03-12T02:21:59.331 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-03-12T02:21:59.482 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-12T02:21:59.624 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-03-12T02:21:59.794 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-12T02:21:59.965 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-03-12T02:22:00.105 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-03-12T02:22:00.265 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-03-12T02:22:00.406 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-03-12T02:22:00.526 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-03-12T02:22:00.659 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-03-12T02:22:00.813 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-03-12T02:22:00.815 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:00.963 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-12T02:22:01.131 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-03-12T02:22:01.428 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-03-12T02:22:01.667 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-03-12T02:22:01.699 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-03-12T02:22:01.701 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 2, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-12T02:22:01.738 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:01 +0000 (0:00:11.278) 0:00:42.025 ********** 2021-03-12T02:22:01.831 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:01.832 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-03-12T02:22:02.003 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-12T02:22:02.183 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-12T02:22:02.185 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-12T02:22:02.330 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-12T02:22:02.341 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-12T02:22:02.468 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-12T02:22:02.479 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-12T02:22:02.627 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-03-12T02:22:02.629 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwa 2021-03-12T02:22:02.630 INFO:teuthology.task.ansible.out:hlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-12T02:22:02.756 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-12T02:22:02.767 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-12T02:22:02.925 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-03-12T02:22:02.927 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-03-12T02:22:02.980 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-12T02:22:03.074 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-03-12T02:22:03.190 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-12T02:22:03.224 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-03-12T02:22:03.331 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-03-12T02:22:03.373 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-03-12T02:22:03.376 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:03.397 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:03.482 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-03-12T02:22:03.484 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-03-12T02:22:03.546 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-12T02:22:03.547 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-03-12T02:22:03.623 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-12T02:22:03.625 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-03-12T02:22:03.687 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-12T02:22:03.689 INFO:teuthology.task.ansible.out:son", "validate_certs": true} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:03.698 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:03.762 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-12T02:22:03.837 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-03-12T02:22:03.902 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-12T02:22:03.980 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-12T02:22:03.982 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-12T02:22:04.042 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-12T02:22:04.116 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-12T02:22:04.180 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-03-12T02:22:04.254 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-12T02:22:04.256 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-03-12T02:22:04.308 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-03-12T02:22:04.392 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-03-12T02:22:04.457 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-03-12T02:22:04.532 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-12T02:22:04.585 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-12T02:22:04.660 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-12T02:22:04.724 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-03-12T02:22:04.726 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-03-12T02:22:04.810 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-03-12T02:22:04.811 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-03-12T02:22:04.873 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-12T02:22:04.875 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:04.958 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-12T02:22:04.960 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-03-12T02:22:05.065 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-12T02:22:05.109 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-12T02:22:05.238 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-03-12T02:22:05.272 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-12T02:22:05.406 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-12T02:22:05.408 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-12T02:22:05.410 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-12T02:22:05.536 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-12T02:22:05.569 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-12T02:22:05.701 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-03-12T02:22:05.712 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-12T02:22:05.845 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-03-12T02:22:05.847 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-12T02:22:05.868 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-12T02:22:05.998 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-12T02:22:06.042 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-12T02:22:06.166 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-03-12T02:22:06.168 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:06.204 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-12T02:22:06.205 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:06.389 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-12T02:22:06.391 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:06.401 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-12T02:22:06.403 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-03-12T02:22:06.570 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-12T02:22:07.424 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "e 2021-03-12T02:22:07.426 INFO:teuthology.task.ansible.out:xclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, 2021-03-12T02:22:07.427 INFO:teuthology.task.ansible.out:"exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", 2021-03-12T02:22:07.429 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7Ee 2021-03-12T02:22:07.431 INFO:teuthology.task.ansible.out:mIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1 2021-03-12T02:22:07.432 INFO:teuthology.task.ansible.out:qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-12T02:22:07.526 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-12T02:22:07.560 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-12T02:22:07.693 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-12T02:22:07.725 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-12T02:22:07.872 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-12T02:22:07.874 INFO:teuthology.task.ansible.out: ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-12T02:22:08.019 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-12T02:22:08.021 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:08.061 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-12T02:22:08.064 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:08.213 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-12T02:22:08.214 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:08.269 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-12T02:22:08.271 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-12T02:22:08.344 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-12T02:22:08.346 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:08.432 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-12T02:22:08.434 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-12T02:22:08.491 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-12T02:22:08.594 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-12T02:22:08.614 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-12T02:22:08.753 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-12T02:22:08.766 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-12T02:22:08.891 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-12T02:22:08.980 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-03-12T02:22:08.982 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-03-12T02:22:09.065 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-12T02:22:09.067 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-12T02:22:09.140 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-12T02:22:09.204 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-12T02:22:09.206 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-12T02:22:09.208 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-12T02:22:09.298 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-12T02:22:09.374 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-12T02:22:09.448 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-12T02:22:09.526 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-12T02:22:09.601 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-12T02:22:09.676 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-12T02:22:09.751 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-12T02:22:09.829 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-12T02:22:09.887 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-03-12T02:22:09.962 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-12T02:22:10.022 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-12T02:22:10.107 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-12T02:22:10.113 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-03-12T02:22:10.171 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-12T02:22:10.253 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-12T02:22:10.255 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:10.350 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-12T02:22:10.414 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-12T02:22:10.489 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-12T02:22:10.553 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-12T02:22:10.555 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-03-12T02:22:10.626 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-12T02:22:10.691 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-12T02:22:10.770 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-12T02:22:10.834 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-12T02:22:10.921 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-12T02:22:10.965 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-12T02:22:10.966 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-03-12T02:22:11.107 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-12T02:22:11.109 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-03-12T02:22:11.255 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-12T02:22:11.256 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-12T02:22:11.256 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:11.387 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-03-12T02:22:11.389 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-12T02:22:11.545 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", " 2021-03-12T02:22:11.547 INFO:teuthology.task.ansible.out:key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-03-12T02:22:11.567 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-12T02:22:11.704 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-12T02:22:11.799 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-03-12T02:22:11.842 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:11 +0000 (0:00:10.104) 0:00:52.130 ********** 2021-03-12T02:22:11.916 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-03-12T02:22:11.992 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-12T02:22:11.993 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.003 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.046 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:12.048 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@de 2021-03-12T02:22:12.049 INFO:teuthology.task.ansible.out:ll r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.069 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.090 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.101 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.131 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.236 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:12.238 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-03-12T02:22:12.323 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-12T02:22:12.324 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-03-12T02:22:12.337 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.348 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.411 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:12.413 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-03-12T02:22:12.486 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-03-12T02:22:12.540 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-03-12T02:22:12.615 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-03-12T02:22:12.626 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.637 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.639 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:12.650 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.661 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.672 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.673 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-03-12T02:22:12.689 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-12T02:22:12.691 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.693 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.705 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.722 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.726 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.729 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.741 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.752 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.754 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.763 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.774 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.776 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.786 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.797 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.799 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.809 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.810 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-12T02:22:12.822 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.824 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.837 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.838 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:12.849 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.860 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.871 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.883 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.884 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.894 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.905 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.907 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.917 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.919 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.928 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.942 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => 2021-03-12T02:22:12.943 INFO:teuthology.task.ansible.out: {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.953 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.964 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.987 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:12.998 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.009 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.021 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.022 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:13.033 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.036 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:13.045 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.046 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.059 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.070 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.081 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:13.082 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0 2021-03-12T02:22:13.084 INFO:teuthology.task.ansible.out:ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.093 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.115 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.117 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.137 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.149 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.170 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.172 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.182 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.193 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.204 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.216 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.228 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:13.230 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.241 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.264 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.277 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.278 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.289 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.302 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.304 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:13.326 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.328 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.339 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.349 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.360 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.371 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.383 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.385 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e8 2021-03-12T02:22:13.386 INFO:teuthology.task.ansible.out:2e"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.398 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.410 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.426 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.439 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.460 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.471 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.483 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.494 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.496 INFO:teuthology.task.ansible.out: skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.508 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.509 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:13.520 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:13.522 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-12T02:22:13.534 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.555 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.559 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.567 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.578 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.580 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.590 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.601 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.602 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.613 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.625 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.636 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.647 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.649 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-12T02:22:13.660 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-12T02:22:13.662 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.674 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.686 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.687 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:13.687 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.699 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.699 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:13.701 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.711 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.722 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.724 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.733 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.735 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.745 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.747 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.802 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:13 +0000 (0:00:01.960) 0:00:54.090 ********** 2021-03-12T02:22:13.818 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:13.820 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-12T02:22:13.890 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-03-12T02:22:13.892 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:13.913 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:13 +0000 (0:00:00.111) 0:00:54.202 ********** 2021-03-12T02:22:13.946 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:13.948 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-12T02:22:14.107 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-03-12T02:22:14.109 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-12T02:22:14.166 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:14.168 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-03-12T02:22:14.241 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:14.243 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-12T02:22:14.295 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:14.296 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-03-12T02:22:14.369 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:14.371 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-12T02:22:14.443 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi077.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-12T02:22:14.509 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:14.510 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-12T02:22:14.594 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:14.596 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-03-12T02:22:14.639 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-03-12T02:22:14.744 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:14.746 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-12T02:22:14.777 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc 2021-03-12T02:22:14.779 INFO:teuthology.task.ansible.out:+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-03-12T02:22:14.903 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:14.905 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-12T02:22:14.915 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:14.916 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-12T02:22:14.918 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-03-12T02:22:15.043 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-12T02:22:15.055 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:15.057 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-12T02:22:15.170 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:15.171 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-12T02:22:15.236 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:15.238 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-03-12T02:22:15.331 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:15.333 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-12T02:22:15.385 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:15.386 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-03-12T02:22:15.491 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:15.492 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-03-12T02:22:15.528 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi077.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-12T02:22:15.630 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:15.632 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-03-12T02:22:15.684 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:15.687 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-12T02:22:15.800 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:15.802 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-03-12T02:22:15.843 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:15.845 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-12T02:22:15.940 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-12T02:22:15.994 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:15.996 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-12T02:22:16.069 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q 2021-03-12T02:22:16.071 INFO:teuthology.task.ansible.out:+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.local", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-03-12T02:22:16.167 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:16.169 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-12T02:22:16.220 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:16.222 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-12T02:22:16.321 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:16.323 INFO:teuthology.task.ansible.out:TASK [Remove revoked users] **************************************************** ok: [smithi077.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-12T02:22:16.374 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:16.376 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-12T02:22:16.492 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:16.494 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-12T02:22:16.559 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:16.561 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-03-12T02:22:16.681 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:16.683 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-12T02:22:16.713 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:16.715 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-12T02:22:16.843 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:16.845 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-12T02:22:16.890 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-12T02:22:16.988 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:16.990 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-12T02:22:17.062 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:17.064 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-03-12T02:22:17.117 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:17.119 INFO:teuthology.task.ansible.out:TASK [Remove revoked users] **************************************************** ok: [smithi077.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-12T02:22:17.225 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:17.228 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-03-12T02:22:17.229 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-12T02:22:17.279 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:17.280 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-12T02:22:17.395 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:17.397 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-12T02:22:17.427 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:17.429 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-12T02:22:17.533 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:17.534 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-03-12T02:22:17.586 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:17.587 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-12T02:22:17.660 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:17.662 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-12T02:22:17.728 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi077.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-12T02:22:17.793 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:17.795 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-12T02:22:17.883 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi077.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-12T02:22:17.929 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-03-12T02:22:18.026 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi077.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-12T02:22:18.079 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:18.080 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-03-12T02:22:18.164 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:18.166 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-12T02:22:18.219 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-03-12T02:22:18.315 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:18.317 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-12T02:22:18.348 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:18.349 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-03-12T02:22:18.454 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi077.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-12T02:22:18.508 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-03-12T02:22:18.594 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:18.596 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-12T02:22:18.669 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:18.671 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-12T02:22:18.746 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi077.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-12T02:22:18.825 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-12T02:22:18.884 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:18.886 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-12T02:22:18.969 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:18.971 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-12T02:22:19.025 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi077.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-12T02:22:19.099 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:19.101 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-12T02:22:19.174 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:19.176 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-12T02:22:19.234 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:19.236 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-12T02:22:19.332 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:19.334 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-12T02:22:19.375 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:19.377 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-12T02:22:19.471 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:19.473 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-12T02:22:19.503 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:19.505 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-12T02:22:19.609 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:19.610 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-12T02:22:19.641 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:19.643 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-12T02:22:19.756 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:19.758 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-03-12T02:22:19.767 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:19.769 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-12T02:22:19.905 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi077.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AA 2021-03-12T02:22:19.907 INFO:teuthology.task.ansible.out:AAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjLd+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-03-12T02:22:20.047 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-12T02:22:20.057 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:20.059 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-12T02:22:20.185 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:20.187 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-12T02:22:20.207 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:20.209 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-03-12T02:22:20.325 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-12T02:22:20.356 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:20.357 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-12T02:22:20.465 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:20.467 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-12T02:22:20.539 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:20.541 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-12T02:22:20.592 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:20.594 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-12T02:22:20.687 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:20.689 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-12T02:22:20.751 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:20.752 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-03-12T02:22:20.835 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:20.837 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-12T02:22:20.888 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-12T02:22:20.983 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:20.985 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-12T02:22:21.026 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:21.029 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-12T02:22:21.126 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:21.127 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-12T02:22:21.158 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:21.160 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-12T02:22:21.277 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:21.281 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-12T02:22:21.406 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-12T02:22:21.428 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:21.430 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-12T02:22:21.493 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:21 +0000 (0:00:07.579) 0:01:01.781 ********** 2021-03-12T02:22:21.574 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:21.576 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-12T02:22:21.746 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-12T02:22:21.946 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-12T02:22:22.093 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-12T02:22:22.146 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-03-12T02:22:22.148 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615515741.6354892-17116-75136146989872/source", "state": "file", "uid": 0} 2021-03-12T02:22:22.182 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:22 +0000 (0:00:00.688) 0:01:02.470 ********** 2021-03-12T02:22:22.226 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:22.228 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-12T02:22:22.300 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-03-12T02:22:22.302 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => { "msg": "Host smithi077.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-03-12T02:22:22.325 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:22 +0000 (0:00:00.143) 0:01:02.614 ********** 2021-03-12T02:22:22.353 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:22.355 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", " 2021-03-12T02:22:22.357 INFO:teuthology.task.ansible.out:key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-03-12T02:22:22.491 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-03-12T02:22:22.492 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-03-12T02:22:22.517 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:22 +0000 (0:00:00.191) 0:01:02.805 ********** 2021-03-12T02:22:22.536 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-12T02:22:22.537 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-12T02:22:22.672 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:22 +0000 (0:00:00.155) 0:01:02.960 ********** 2021-03-12T02:22:22.678 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-03-12T02:22:22.680 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:22.713 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:22 +0000 (0:00:00.041) 0:01:03.002 ********** 2021-03-12T02:22:22.753 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-12T02:22:22.754 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:22.767 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:22.778 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:22.780 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:22.802 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:22 +0000 (0:00:00.088) 0:01:03.090 ********** 2021-03-12T02:22:22.804 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:22.811 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-12T02:22:22.812 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:22.814 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:22.816 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:22.817 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:22.820 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:22.831 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:22.873 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-03-12T02:22:22.875 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:22.901 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:22 +0000 (0:00:00.098) 0:01:03.189 ********** 2021-03-12T02:22:22.983 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-03-12T02:22:22.985 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smith 2021-03-12T02:22:22.987 INFO:teuthology.task.ansible.out:farm", "validate_certs": true} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.007 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:23 +0000 (0:00:00.106) 0:01:03.296 ********** 2021-03-12T02:22:23.014 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.078 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-03-12T02:22:23.080 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.106 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:23 +0000 (0:00:00.098) 0:01:03.395 ********** 2021-03-12T02:22:23.124 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:23.126 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-03-12T02:22:23.181 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-03-12T02:22:23.183 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.204 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:23 +0000 (0:00:00.098) 0:01:03.493 ********** 2021-03-12T02:22:23.246 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi114.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-03-12T02:22:23.255 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.277 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.298 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-03-12T02:22:23.302 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.330 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:23 +0000 (0:00:00.125) 0:01:03.618 ********** 2021-03-12T02:22:23.350 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.351 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.358 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.359 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.370 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.381 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.382 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.393 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.404 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.415 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.417 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.431 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} TASK [common : Set entitlements_path] ****************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", 2021-03-12T02:22:23.442 INFO:teuthology.task.ansible.out:"ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.466 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:23 +0000 (0:00:00.136) 0:01:03.755 ********** 2021-03-12T02:22:23.482 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.489 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.493 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.494 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.495 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.496 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.521 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.533 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.546 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.557 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.568 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.570 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.580 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.591 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.602 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.624 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi077.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-12T02:22:23.625 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.668 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:23 +0000 (0:00:00.201) 0:01:03.957 ********** 2021-03-12T02:22:23.678 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.679 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.680 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.681 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.682 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.683 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.700 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.701 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.712 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.714 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.724 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.725 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.736 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.738 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:23.751 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.753 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.763 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.775 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.777 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.805 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:23 +0000 (0:00:00.136) 0:01:04.093 ********** 2021-03-12T02:22:23.825 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.827 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.827 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.838 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.840 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.850 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.861 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.872 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.883 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.894 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.896 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.917 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:23 +0000 (0:00:00.112) 0:01:04.205 ********** 2021-03-12T02:22:23.925 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-12T02:22:23.928 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'n 2021-03-12T02:22:23.929 INFO:teuthology.task.ansible.out:ame': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.940 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.950 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.961 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.972 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.974 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.984 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:23.995 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.007 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-03-12T02:22:24.008 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.031 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:24 +0000 (0:00:00.113) 0:01:04.319 ********** 2021-03-12T02:22:24.039 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.046 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.052 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.074 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.085 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.088 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.096 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.098 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.108 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.119 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.121 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.157 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:24 +0000 (0:00:00.113) 0:01:04.433 ********** 2021-03-12T02:22:24.168 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432a 2021-03-12T02:22:24.169 INFO:teuthology.task.ansible.out:ac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.174 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.189 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.191 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.201 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.203 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.217 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.224 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.236 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.238 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.248 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.259 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.262 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.271 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.282 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.284 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi077.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-12T02:22:24.315 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:24 +0000 (0:00:00.170) 0:01:04.603 ********** 2021-03-12T02:22:24.324 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-12T02:22:24.329 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.331 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.331 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.331 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.333 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.343 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.345 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.356 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.436 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:24 +0000 (0:00:00.121) 0:01:04.724 ********** 2021-03-12T02:22:24.452 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-03-12T02:22:24.454 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.493 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:24 +0000 (0:00:00.057) 0:01:04.782 ********** 2021-03-12T02:22:24.536 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-03-12T02:22:24.538 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.566 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:24 +0000 (0:00:00.072) 0:01:04.854 ********** 2021-03-12T02:22:24.580 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-03-12T02:22:24.581 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.609 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:24 +0000 (0:00:00.042) 0:01:04.896 ********** 2021-03-12T02:22:24.718 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.744 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:24 +0000 (0:00:00.135) 0:01:05.032 ********** 2021-03-12T02:22:24.814 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-12T02:22:24.818 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:24.820 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi114.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-03-12T02:22:24.846 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:24 +0000 (0:00:00.102) 0:01:05.134 ********** 2021-03-12T02:22:24.956 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-03-12T02:22:24.958 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-12T02:22:24.980 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:24 +0000 (0:00:00.133) 0:01:05.268 ********** 2021-03-12T02:22:24.986 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:24.988 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-03-12T02:22:25.064 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:25.088 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:25 +0000 (0:00:00.108) 0:01:05.376 ********** 2021-03-12T02:22:25.115 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:25.117 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-12T02:22:25.190 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-03-12T02:22:25.191 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:25.240 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:25 +0000 (0:00:00.152) 0:01:05.529 ********** 2021-03-12T02:22:25.259 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:25.261 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-03-12T02:22:25.334 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-03-12T02:22:25.336 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:25.360 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:25 +0000 (0:00:00.119) 0:01:05.648 ********** 2021-03-12T02:22:25.384 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:25.386 INFO:teuthology.task.ansible.out:TASK [Remove revoked users] **************************************************** ok: [smithi114.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-12T02:22:25.481 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-03-12T02:22:25.483 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:25.505 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:25 +0000 (0:00:00.145) 0:01:05.794 ********** 2021-03-12T02:22:25.524 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:25.526 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-12T02:22:25.579 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-03-12T02:22:25.581 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:25.604 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:25 +0000 (0:00:00.098) 0:01:05.892 ********** 2021-03-12T02:22:25.652 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:25.654 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-12T02:22:25.675 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-03-12T02:22:25.676 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:25.707 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:25 +0000 (0:00:00.103) 0:01:05.995 ********** 2021-03-12T02:22:25.816 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:25.818 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-03-12T02:22:25.888 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-03-12T02:22:25.890 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-12T02:22:25.913 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:25 +0000 (0:00:00.205) 0:01:06.201 ********** 2021-03-12T02:22:26.027 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:26.029 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-03-12T02:22:26.103 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:26.170 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:26 +0000 (0:00:00.256) 0:01:06.458 ********** 2021-03-12T02:22:26.283 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:26.285 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-12T02:22:26.382 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-03-12T02:22:26.383 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:26.430 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:26 +0000 (0:00:00.259) 0:01:06.718 ********** 2021-03-12T02:22:26.484 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:26.485 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-12T02:22:26.539 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-03-12T02:22:26.542 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:26.583 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:26 +0000 (0:00:00.146) 0:01:06.864 ********** 2021-03-12T02:22:26.631 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:26.633 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-12T02:22:26.718 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-03-12T02:22:26.720 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:26.748 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:26 +0000 (0:00:00.172) 0:01:07.037 ********** 2021-03-12T02:22:26.831 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:26.834 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-12T02:22:26.866 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* skipping: [smithi077.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-12T02:22:26.889 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:26 +0000 (0:00:00.140) 0:01:07.177 ********** 2021-03-12T02:22:27.011 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-12T02:22:27.013 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [Remove revoked users] **************************************************** ok: [smithi114.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-12T02:22:27.035 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:27 +0000 (0:00:00.145) 0:01:07.323 ********** 2021-03-12T02:22:27.160 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-12T02:22:27.303 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-12T02:22:27.490 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-12T02:22:27.492 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:27.588 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-12T02:22:27.590 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.287896", "end": "2021-03-12 02:22:28.038309", "rc": 0, "start": "2021-03-12 02:22:27.750413", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-12T02:22:27.617 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:27 +0000 (0:00:00.581) 0:01:07.905 ********** 2021-03-12T02:22:27.625 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:27.670 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:27.671 INFO:teuthology.task.ansible.out:TASK [Remove revoked users] **************************************************** ok: [smithi114.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-12T02:22:27.770 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:27.811 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:27 +0000 (0:00:00.194) 0:01:08.099 ********** 2021-03-12T02:22:27.837 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:27.839 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-12T02:22:27.906 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:27.908 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:27.932 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:27 +0000 (0:00:00.121) 0:01:08.221 ********** 2021-03-12T02:22:27.934 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:27.984 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi114.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-12T02:22:28.060 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:28.062 INFO:teuthology.task.ansible.out:TASK [common : Mask sleep units] *********************************************** skipping: [smithi077.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-03-12T02:22:28.072 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-03-12T02:22:28.074 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:28.084 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-03-12T02:22:28.086 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:28.109 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:28 +0000 (0:00:00.176) 0:01:08.397 ********** 2021-03-12T02:22:28.128 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:28.129 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-12T02:22:28.205 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-12T02:22:28.207 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:28.232 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:28 +0000 (0:00:00.122) 0:01:08.520 ********** 2021-03-12T02:22:28.257 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:28.259 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-12T02:22:28.313 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-03-12T02:22:28.314 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:28.351 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:28 +0000 (0:00:00.119) 0:01:08.640 ********** 2021-03-12T02:22:28.407 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:28.409 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-12T02:22:28.478 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:28.497 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:28 +0000 (0:00:00.145) 0:01:08.785 ********** 2021-03-12T02:22:28.499 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:28.558 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:28.560 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-12T02:22:28.570 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-03-12T02:22:28.571 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:28.595 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:28 +0000 (0:00:00.098) 0:01:08.883 ********** 2021-03-12T02:22:28.669 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-03-12T02:22:28.671 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:28.696 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:28 +0000 (0:00:00.100) 0:01:08.984 ********** 2021-03-12T02:22:28.702 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:28.703 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-12T02:22:28.815 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-12T02:22:28.953 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-12T02:22:29.038 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-03-12T02:22:29.040 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.205723", "end": "2021-03-12 02:22:29.500384", "rc": 0, "start": "2021-03-12 02:22:29.294661", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:22:29.069 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:29 +0000 (0:00:00.370) 0:01:09.355 ********** 2021-03-12T02:22:29.100 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:29.106 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-12T02:22:29.143 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-03-12T02:22:29.145 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:29.166 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:29 +0000 (0:00:00.099) 0:01:09.455 ********** 2021-03-12T02:22:29.246 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-12T02:22:29.248 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-12T02:22:29.398 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ [WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.023351", "end": "2021-03-12 02:22:29.853094", "rc": 0, "start": "2021-03-12 02:22:29.829743", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [Remove revoked users] **************************************************** ok: [smithi114.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-12T02:22:29.426 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:29 +0000 (0:00:00.258) 0:01:09.713 ********** 2021-03-12T02:22:29.542 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-12T02:22:29.711 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-12T02:22:29.857 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-12T02:22:29.996 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-12T02:22:30.144 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-12T02:22:30.146 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:30.285 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-12T02:22:30.426 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-12T02:22:30.568 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-03-12T02:22:30.724 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-03-12T02:22:30.876 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-12T02:22:30.878 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:31.036 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-03-12T02:22:31.178 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-12T02:22:31.316 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-12T02:22:31.475 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-12T02:22:31.631 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-12T02:22:31.808 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-12T02:22:32.001 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-12T02:22:32.154 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-03-12T02:22:32.304 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-12T02:22:32.388 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:32 +0000 (0:00:02.963) 0:01:12.676 ********** 2021-03-12T02:22:32.739 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-03-12T02:22:32.741 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1615515667, "cache_updated": false, "changed": false} 2021-03-12T02:22:32.769 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:32 +0000 (0:00:00.380) 0:01:13.057 ********** 2021-03-12T02:22:32.985 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-03-12T02:22:32.987 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615515752.5502489-18858-237073343820402/source", "state": "file", "uid": 0} 2021-03-12T02:22:33.028 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:33 +0000 (0:00:00.259) 0:01:13.317 ********** 2021-03-12T02:22:33.198 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-03-12T02:22:33.200 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => { "msg": "Host smithi114.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-03-12T02:22:33.233 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:33 +0000 (0:00:00.204) 0:01:13.521 ********** 2021-03-12T02:22:33.359 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi114.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-03-12T02:22:33.384 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:33 +0000 (0:00:00.151) 0:01:13.673 ********** 2021-03-12T02:22:33.475 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-03-12T02:22:33.477 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:33.503 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:33 +0000 (0:00:00.114) 0:01:13.788 ********** 2021-03-12T02:22:33.585 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-03-12T02:22:33.587 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:33.608 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:33 +0000 (0:00:00.108) 0:01:13.896 ********** 2021-03-12T02:22:33.687 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-03-12T02:22:33.689 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:33.711 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:33 +0000 (0:00:00.103) 0:01:13.999 ********** 2021-03-12T02:22:33.782 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-03-12T02:22:33.784 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:33.805 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:33 +0000 (0:00:00.093) 0:01:14.093 ********** 2021-03-12T02:22:33.813 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-03-12T02:22:33.815 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"cache_update_time": 1615515667, "cache_updated": false, "changed": false} 2021-03-12T02:22:33.845 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:33 +0000 (0:00:00.040) 0:01:14.133 ********** 2021-03-12T02:22:33.904 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-03-12T02:22:33.906 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:33.939 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:33 +0000 (0:00:00.093) 0:01:14.227 ********** 2021-03-12T02:22:33.951 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-03-12T02:22:33.953 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:33.988 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:33 +0000 (0:00:00.048) 0:01:14.276 ********** 2021-03-12T02:22:34.047 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-03-12T02:22:34.049 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:34.076 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:34 +0000 (0:00:00.087) 0:01:14.363 ********** 2021-03-12T02:22:34.163 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-03-12T02:22:34.165 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:34.189 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:34 +0000 (0:00:00.113) 0:01:14.477 ********** 2021-03-12T02:22:34.287 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-03-12T02:22:34.289 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:34.312 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:34 +0000 (0:00:00.123) 0:01:14.600 ********** 2021-03-12T02:22:34.415 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-03-12T02:22:34.417 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi114.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-12T02:22:34.445 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:34 +0000 (0:00:00.132) 0:01:14.733 ********** 2021-03-12T02:22:34.557 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-03-12T02:22:34.560 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:34.584 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:34 +0000 (0:00:00.139) 0:01:14.872 ********** 2021-03-12T02:22:34.713 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:34.716 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:34.749 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:34 +0000 (0:00:00.164) 0:01:15.037 ********** 2021-03-12T02:22:34.873 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-03-12T02:22:34.875 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-12T02:22:34.900 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:34 +0000 (0:00:00.151) 0:01:15.188 ********** 2021-03-12T02:22:34.986 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-03-12T02:22:34.988 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:35.018 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:35 +0000 (0:00:00.117) 0:01:15.306 ********** 2021-03-12T02:22:35.020 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:35.027 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-03-12T02:22:35.027 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-03-12T02:22:35.055 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:35 +0000 (0:00:00.036) 0:01:15.343 ********** 2021-03-12T02:22:35.093 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-03-12T02:22:35.095 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-12T02:22:35.127 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:35 +0000 (0:00:00.071) 0:01:15.415 ********** 2021-03-12T02:22:35.220 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-03-12T02:22:35.227 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:35.247 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:35 +0000 (0:00:00.120) 0:01:15.535 ********** 2021-03-12T02:22:35.354 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-03-12T02:22:35.356 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:35.380 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:35 +0000 (0:00:00.133) 0:01:15.668 ********** 2021-03-12T02:22:35.464 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-03-12T02:22:35.466 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:35.488 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:35 +0000 (0:00:00.107) 0:01:15.776 ********** 2021-03-12T02:22:35.578 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-12T02:22:35.580 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:35.607 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:35 +0000 (0:00:00.115) 0:01:15.891 ********** 2021-03-12T02:22:35.694 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-03-12T02:22:35.696 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-12T02:22:35.719 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:35 +0000 (0:00:00.115) 0:01:16.007 ********** 2021-03-12T02:22:35.799 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-12T02:22:35.801 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:35.822 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:35 +0000 (0:00:00.102) 0:01:16.110 ********** 2021-03-12T02:22:35.893 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-03-12T02:22:35.895 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:35.923 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:35 +0000 (0:00:00.101) 0:01:16.211 ********** 2021-03-12T02:22:36.013 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-03-12T02:22:36.015 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"cache_update_time": 1615515667, "cache_updated": false, "changed": false} TASK [common : Store list of enabled repos] ************************************ skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:36.038 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:36 +0000 (0:00:00.114) 0:01:16.326 ********** 2021-03-12T02:22:36.061 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:36 +0000 (0:00:00.023) 0:01:16.349 ********** 2021-03-12T02:22:36.162 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-03-12T02:22:36.165 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:36.188 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:36 +0000 (0:00:00.127) 0:01:16.476 ********** 2021-03-12T02:22:36.273 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-03-12T02:22:36.274 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:36.297 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:36 +0000 (0:00:00.108) 0:01:16.585 ********** 2021-03-12T02:22:36.375 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-03-12T02:22:36.377 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:36.411 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:36 +0000 (0:00:00.113) 0:01:16.699 ********** 2021-03-12T02:22:36.497 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:36.498 INFO:teuthology.task.ansible.out:TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi114.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-12T02:22:36.500 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:36.528 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:36 +0000 (0:00:00.117) 0:01:16.816 ********** 2021-03-12T02:22:36.548 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-12T02:22:36.549 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-03-12T02:22:36.623 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:36.647 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:36 +0000 (0:00:00.119) 0:01:16.935 ********** 2021-03-12T02:22:36.715 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-03-12T02:22:36.717 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:36.743 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:36 +0000 (0:00:00.096) 0:01:17.032 ********** 2021-03-12T02:22:36.822 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:36.848 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:36 +0000 (0:00:00.104) 0:01:17.136 ********** 2021-03-12T02:22:36.900 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-12T02:22:36.902 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-03-12T02:22:36.933 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-03-12T02:22:36.934 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:36.944 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:36.968 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:36 +0000 (0:00:00.120) 0:01:17.257 ********** 2021-03-12T02:22:37.047 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-03-12T02:22:37.049 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-12T02:22:37.071 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:37 +0000 (0:00:00.103) 0:01:17.360 ********** 2021-03-12T02:22:37.150 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:37.175 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:37 +0000 (0:00:00.103) 0:01:17.464 ********** 2021-03-12T02:22:37.202 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-12T02:22:37.204 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-03-12T02:22:37.230 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:37 +0000 (0:00:00.055) 0:01:17.519 ********** 2021-03-12T02:22:37.387 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-03-12T02:22:37.388 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-03-12T02:22:37.411 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:37 +0000 (0:00:00.180) 0:01:17.700 ********** 2021-03-12T02:22:37.462 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-12T02:22:37.464 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.087260", "end": "2021-03-12 02:22:37.298653", "rc": 0, "start": "2021-03-12 02:22:37.211393", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-12T02:22:37.495 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:37 +0000 (0:00:00.083) 0:01:17.783 ********** 2021-03-12T02:22:37.602 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:37.792 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:37 +0000 (0:00:00.130) 0:01:17.914 ********** TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 12 March 2021 02:22:37 +0000 (0:00:00.104) 0:01:18.018 ********** 2021-03-12T02:22:37.820 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-12T02:22:37.821 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-03-12T02:22:37.822 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-12T02:22:37.822 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-03-12T02:22:37.833 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-12T02:22:37.835 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-03-12T02:22:37.869 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:37 +0000 (0:00:00.138) 0:01:18.157 ********** 2021-03-12T02:22:37.958 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-12T02:22:37.960 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:37.984 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:37 +0000 (0:00:00.115) 0:01:18.273 ********** 2021-03-12T02:22:38.054 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:38.078 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:38 +0000 (0:00:00.093) 0:01:18.366 ********** 2021-03-12T02:22:38.105 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-12T02:22:38.107 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-03-12T02:22:38.151 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-03-12T02:22:38.153 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:38.183 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:38 +0000 (0:00:00.104) 0:01:18.471 ********** 2021-03-12T02:22:38.284 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-03-12T02:22:38.286 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:38.310 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:38 +0000 (0:00:00.125) 0:01:18.597 ********** 2021-03-12T02:22:38.388 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-03-12T02:22:38.390 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:38.414 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:38 +0000 (0:00:00.105) 0:01:18.702 ********** 2021-03-12T02:22:38.432 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-12T02:22:38.433 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-03-12T02:22:38.464 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:38 +0000 (0:00:00.050) 0:01:18.752 ********** 2021-03-12T02:22:38.547 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:38.549 INFO:teuthology.task.ansible.out:TASK [common : Include apt_system vars] **************************************** ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-03-12T02:22:38.578 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:38 +0000 (0:00:00.113) 0:01:18.866 ********** 2021-03-12T02:22:38.657 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-03-12T02:22:38.659 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:38.680 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:38 +0000 (0:00:00.102) 0:01:18.969 ********** 2021-03-12T02:22:38.749 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-12T02:22:38.751 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-12T02:22:38.773 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:38 +0000 (0:00:00.092) 0:01:19.061 ********** 2021-03-12T02:22:38.863 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-12T02:22:38.864 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-12T02:22:38.874 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-12T02:22:38.875 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-12T02:22:38.909 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:38 +0000 (0:00:00.136) 0:01:19.197 ********** 2021-03-12T02:22:38.988 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-03-12T02:22:38.990 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:39.011 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:39 +0000 (0:00:00.102) 0:01:19.300 ********** 2021-03-12T02:22:39.662 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-03-12T02:22:39.664 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:01.105852", "end": "2021-03-12 02:22:39.500437", "rc": 0, "start": "2021-03-12 02:22:38.394585", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:22:39.686 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:39 +0000 (0:00:00.674) 0:01:19.975 ********** 2021-03-12T02:22:39.756 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-03-12T02:22:39.758 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:39.781 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:39 +0000 (0:00:00.094) 0:01:20.069 ********** 2021-03-12T02:22:39.969 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-03-12T02:22:39.970 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"cache_update_time": 1615515667, "cache_updated": false, "changed": false} 2021-03-12T02:22:39.995 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:39 +0000 (0:00:00.213) 0:01:20.283 ********** 2021-03-12T02:22:40.002 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-03-12T02:22:40.003 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.022630", "end": "2021-03-12 02:22:39.821162", "rc": 0, "start": "2021-03-12 02:22:39.798532", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:22:40.030 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:40 +0000 (0:00:00.035) 0:01:20.318 ********** 2021-03-12T02:22:40.374 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-03-12T02:22:40.376 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-03-12T02:22:40.399 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:40 +0000 (0:00:00.368) 0:01:20.687 ********** 2021-03-12T02:22:40.854 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-03-12T02:22:40.856 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-03-12T02:22:40.877 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:40 +0000 (0:00:00.478) 0:01:21.166 ********** 2021-03-12T02:22:41.107 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-12T02:22:41.109 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-03-12T02:22:41.223 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-03-12T02:22:41.255 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:41 +0000 (0:00:00.377) 0:01:21.543 ********** 2021-03-12T02:22:41.334 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-03-12T02:22:41.336 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:41.360 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:41 +0000 (0:00:00.105) 0:01:21.649 ********** 2021-03-12T02:22:41.432 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-03-12T02:22:41.433 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-12T02:22:41.457 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:41 +0000 (0:00:00.096) 0:01:21.745 ********** 2021-03-12T02:22:41.536 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-03-12T02:22:41.538 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:41.562 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:41 +0000 (0:00:00.105) 0:01:21.850 ********** 2021-03-12T02:22:42.003 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-03-12T02:22:42.005 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-03-12T02:22:42.033 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:42 +0000 (0:00:00.470) 0:01:22.321 ********** 2021-03-12T02:22:42.781 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-03-12 02:20:24 UTC", "ActiveEnterTimestampMonotonic": "17347348", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target local-fs.target remote-fs.target sysinit.target systemd-journald.socket var-run.mount basic.target network.target systemd-tmpfiles-setup.service nss-lookup.target system.slice -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-03-12 02:20:24 UTC", "AssertTimestampMonotonic": "17346714", "Before": "shutdown.target xdm.service multi-user.target plymouth-quit.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-03-12 02:20:24 UTC", "ConditionTimestampMonotonic": "17346712", "ConfigurationDirectoryMode": "0755", "Confl 2021-03-12T02:22:42.784 INFO:teuthology.task.ansible.out:icts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1163", "ExecMainStartTimestamp": "Fri 2021-03-12 02:20:24 UTC", "ExecMainStartTimestampMonotonic": "17347322", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-03-12 02:20:24 UTC", "InactiveExitTimestampMonotonic": "17347348", "InvocationID": "fd96771f8aca441fbd7ba5ccd10afe7e", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "Kill 2021-03-12T02:22:42.793 INFO:teuthology.task.ansible.out:Mode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1163", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/tmp 2021-03-12T02:22:42.795 INFO:teuthology.task.ansible.out:/var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-03-12 02:20:24 UTC", "StateChangeTimestampMonotonic": "17347348", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-03-12 02:20:24 UTC", "WatchdogTimestampMonotonic": "17347348", "WatchdogUSec": "0"}} 2021-03-12T02:22:42.809 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:42 +0000 (0:00:00.776) 0:01:23.097 ********** 2021-03-12T02:22:42.900 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-03-12T02:22:42.901 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:42.924 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:42 +0000 (0:00:00.114) 0:01:23.212 ********** 2021-03-12T02:22:43.005 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-03-12T02:22:43.007 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1615515672, "cache_updated": false, "changed": false} 2021-03-12T02:22:43.030 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:43 +0000 (0:00:00.106) 0:01:23.318 ********** 2021-03-12T02:22:43.038 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-03-12T02:22:43.039 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:43.076 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:43 +0000 (0:00:00.045) 0:01:23.364 ********** 2021-03-12T02:22:43.182 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:43.228 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:43 +0000 (0:00:00.151) 0:01:23.516 ********** 2021-03-12T02:22:43.298 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-03-12T02:22:43.300 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:43.325 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:43 +0000 (0:00:00.097) 0:01:23.614 ********** 2021-03-12T02:22:43.397 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-03-12T02:22:43.399 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:43.422 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:43 +0000 (0:00:00.096) 0:01:23.710 ********** 2021-03-12T02:22:43.492 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-03-12T02:22:43.494 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:43.516 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:43 +0000 (0:00:00.093) 0:01:23.804 ********** 2021-03-12T02:22:43.587 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-03-12T02:22:43.589 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:43.612 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:43 +0000 (0:00:00.096) 0:01:23.900 ********** 2021-03-12T02:22:43.705 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-03-12T02:22:43.706 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:43.730 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:43 +0000 (0:00:00.117) 0:01:24.018 ********** 2021-03-12T02:22:43.819 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-03-12T02:22:43.821 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:43.847 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:43 +0000 (0:00:00.117) 0:01:24.135 ********** 2021-03-12T02:22:43.926 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-03-12T02:22:43.928 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:43.951 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:43 +0000 (0:00:00.103) 0:01:24.239 ********** 2021-03-12T02:22:43.972 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-03-12T02:22:43.973 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"cache_update_time": 1615515672, "cache_updated": false, "changed": false} 2021-03-12T02:22:43.974 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:44.000 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:43 +0000 (0:00:00.048) 0:01:24.288 ********** 2021-03-12T02:22:44.060 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-03-12T02:22:44.062 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-03-12T02:22:44.087 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:44 +0000 (0:00:00.087) 0:01:24.375 ********** 2021-03-12T02:22:44.095 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-03-12T02:22:44.097 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:44.123 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:44 +0000 (0:00:00.035) 0:01:24.411 ********** 2021-03-12T02:22:44.171 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-03-12T02:22:44.173 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-03-12T02:22:44.201 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:44 +0000 (0:00:00.078) 0:01:24.489 ********** 2021-03-12T02:22:44.313 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:44.315 INFO:teuthology.task.ansible.out:TASK [testnode : Including major version specific variables.] ****************** ok: [smithi077.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-03-12T02:22:44.348 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:44 +0000 (0:00:00.146) 0:01:24.636 ********** 2021-03-12T02:22:44.480 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-03-12T02:22:44.482 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-12T02:22:44.505 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:44 +0000 (0:00:00.157) 0:01:24.793 ********** 2021-03-12T02:22:44.617 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-03-12T02:22:44.619 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-03-12T02:22:44.648 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:44 +0000 (0:00:00.142) 0:01:24.936 ********** 2021-03-12T02:22:44.677 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:44.681 INFO:teuthology.task.ansible.out:TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-03-12T02:22:44.683 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-12T02:22:44.713 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:44 +0000 (0:00:00.064) 0:01:25.001 ********** 2021-03-12T02:22:44.946 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-03-12T02:22:44.948 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-12T02:22:44.972 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:44 +0000 (0:00:00.259) 0:01:25.261 ********** 2021-03-12T02:22:45.053 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:45.076 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:45 +0000 (0:00:00.103) 0:01:25.364 ********** 2021-03-12T02:22:45.166 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-03-12T02:22:45.168 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:45.192 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:45 +0000 (0:00:00.116) 0:01:25.480 ********** 2021-03-12T02:22:45.559 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-03-12T02:22:45.560 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-03-12T02:22:45.583 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:45 +0000 (0:00:00.391) 0:01:25.871 ********** 2021-03-12T02:22:45.600 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-03-12T02:22:45.603 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"cache_update_time": 1615515672, "cache_updated": false, "changed": false} 2021-03-12T02:22:45.626 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:45 +0000 (0:00:00.042) 0:01:25.914 ********** 2021-03-12T02:22:46.392 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-12T02:22:46.393 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-03-12T02:22:46.457 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-12T02:22:46.459 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-03-12T02:22:46.487 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:46 +0000 (0:00:00.861) 0:01:26.775 ********** 2021-03-12T02:22:46.741 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-12T02:22:46.742 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-03-12T02:22:46.752 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi077"}, "changed": false} 2021-03-12T02:22:46.897 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:46 +0000 (0:00:00.409) 0:01:27.185 ********** 2021-03-12T02:22:47.075 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-12T02:22:47.077 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-03-12T02:22:47.122 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:47 +0000 (0:00:00.222) 0:01:27.408 ********** 2021-03-12T02:22:47.293 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-03-12T02:22:47.295 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-03-12T02:22:47.345 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:47 +0000 (0:00:00.225) 0:01:27.633 ********** 2021-03-12T02:22:47.364 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:48.029 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-12T02:22:48.031 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-03-12T02:22:48.434 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-03-12T02:22:48.509 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-03-12T02:22:48.511 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi077", "ansible_hostname": "smithi077", "ansible_nodename": "smithi077"}, "changed": false, "name": "smithi077"} 2021-03-12T02:22:48.537 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:48 +0000 (0:00:01.191) 0:01:28.825 ********** 2021-03-12T02:22:48.789 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-12T02:22:48.792 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-03-12T02:22:48.830 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:48 +0000 (0:00:00.293) 0:01:29.118 ********** 2021-03-12T02:22:48.978 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-12T02:22:48.980 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-03-12T02:22:49.007 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:49 +0000 (0:00:00.177) 0:01:29.296 ********** 2021-03-12T02:22:49.060 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-03-12T02:22:49.061 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-03-12T02:22:49.103 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:49 +0000 (0:00:00.095) 0:01:29.392 ********** 2021-03-12T02:22:49.110 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-03-12T02:22:49.112 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:49.143 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:49 +0000 (0:00:00.039) 0:01:29.431 ********** 2021-03-12T02:22:49.234 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-12T02:22:49.236 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-12T02:22:49.262 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:49 +0000 (0:00:00.117) 0:01:29.549 ********** 2021-03-12T02:22:49.382 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-12T02:22:49.385 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-12T02:22:49.406 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-12T02:22:49.408 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-12T02:22:49.417 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-12T02:22:49.446 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:49 +0000 (0:00:00.184) 0:01:29.734 ********** 2021-03-12T02:22:49.547 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-03-12T02:22:49.549 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:49.573 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:49 +0000 (0:00:00.126) 0:01:29.861 ********** 2021-03-12T02:22:49.588 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-03-12T02:22:49.626 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:49 +0000 (0:00:00.052) 0:01:29.914 ********** 2021-03-12T02:22:50.106 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-03-12T02:22:50.108 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-12T02:22:50.135 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:50 +0000 (0:00:00.508) 0:01:30.423 ********** 2021-03-12T02:22:50.241 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:50.243 INFO:teuthology.task.ansible.out:TASK [testnode : Create remote.conf] ******************************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:50.280 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:50 +0000 (0:00:00.145) 0:01:30.569 ********** 2021-03-12T02:22:50.374 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-12T02:22:50.376 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:50.397 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:50 +0000 (0:00:00.116) 0:01:30.686 ********** 2021-03-12T02:22:50.491 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-12T02:22:50.493 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:50.520 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:50 +0000 (0:00:00.122) 0:01:30.808 ********** 2021-03-12T02:22:50.590 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-12T02:22:50.592 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:50.614 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:50 +0000 (0:00:00.093) 0:01:30.902 ********** 2021-03-12T02:22:50.684 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:50.721 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:50 +0000 (0:00:00.107) 0:01:31.009 ********** 2021-03-12T02:22:50.750 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-03-12T02:22:50.752 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"cache_update_time": 1615515672, "cache_updated": false, "changed": false} 2021-03-12T02:22:50.784 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:50 +0000 (0:00:00.063) 0:01:31.072 ********** 2021-03-12T02:22:50.805 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-12T02:22:50.807 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:50.840 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:50 +0000 (0:00:00.055) 0:01:31.128 ********** 2021-03-12T02:22:50.931 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-12T02:22:50.933 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:50.955 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:50 +0000 (0:00:00.115) 0:01:31.243 ********** 2021-03-12T02:22:51.036 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-03-12T02:22:51.038 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:51.065 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:51 +0000 (0:00:00.109) 0:01:31.353 ********** 2021-03-12T02:22:51.144 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-03-12T02:22:51.169 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:51 +0000 (0:00:00.104) 0:01:31.457 ********** 2021-03-12T02:22:51.198 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-03-12T02:22:51.200 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-03-12T02:22:51.234 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:51 +0000 (0:00:00.064) 0:01:31.522 ********** 2021-03-12T02:22:51.252 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-03-12T02:22:51.279 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:51 +0000 (0:00:00.045) 0:01:31.567 ********** 2021-03-12T02:22:51.362 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-03-12T02:22:51.390 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:51 +0000 (0:00:00.111) 0:01:31.679 ********** 2021-03-12T02:22:51.477 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-03-12T02:22:51.479 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:51.502 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:51 +0000 (0:00:00.111) 0:01:31.790 ********** 2021-03-12T02:22:51.617 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-03-12T02:22:51.619 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-12T02:22:51.644 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:51 +0000 (0:00:00.142) 0:01:31.933 ********** 2021-03-12T02:22:51.670 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-03-12T02:22:51.671 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-03-12T02:22:51.709 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:51 +0000 (0:00:00.064) 0:01:31.997 ********** 2021-03-12T02:22:51.752 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-03-12T02:22:51.754 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:51.804 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:51 +0000 (0:00:00.094) 0:01:32.092 ********** 2021-03-12T02:22:51.931 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-12T02:22:51.933 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:51.956 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:51 +0000 (0:00:00.152) 0:01:32.244 ********** 2021-03-12T02:22:51.964 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-12T02:22:51.966 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-03-12T02:22:52.045 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:52.085 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:52 +0000 (0:00:00.128) 0:01:32.373 ********** 2021-03-12T02:22:52.098 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ ok: [smithi114.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-03-12T02:22:52.126 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:52 +0000 (0:00:00.041) 0:01:32.415 ********** 2021-03-12T02:22:52.165 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:52.214 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:52 +0000 (0:00:00.087) 0:01:32.502 ********** 2021-03-12T02:22:52.228 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-03-12T02:22:52.229 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:52.266 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:52 +0000 (0:00:00.051) 0:01:32.554 ********** 2021-03-12T02:22:52.317 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-03-12T02:22:52.319 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:52.349 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:52 +0000 (0:00:00.083) 0:01:32.637 ********** 2021-03-12T02:22:52.366 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-03-12T02:22:52.368 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-12T02:22:52.395 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:52 +0000 (0:00:00.044) 0:01:32.682 ********** 2021-03-12T02:22:52.470 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:52.495 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:52 +0000 (0:00:00.101) 0:01:32.783 ********** 2021-03-12T02:22:52.498 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:52.513 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:52.515 INFO:teuthology.task.ansible.out:TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-03-12T02:22:52.516 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:52.566 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:52 +0000 (0:00:00.070) 0:01:32.854 ********** 2021-03-12T02:22:52.625 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-12T02:22:52.627 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:52.652 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:52 +0000 (0:00:00.084) 0:01:32.938 ********** 2021-03-12T02:22:52.760 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-12T02:22:52.762 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:52.784 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:52 +0000 (0:00:00.133) 0:01:33.072 ********** 2021-03-12T02:22:52.874 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-12T02:22:52.875 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:52.906 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:52 +0000 (0:00:00.121) 0:01:33.194 ********** 2021-03-12T02:22:53.019 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-03-12T02:22:53.021 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:53.045 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:53 +0000 (0:00:00.138) 0:01:33.333 ********** 2021-03-12T02:22:53.072 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-03-12T02:22:53.073 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-03-12T02:22:53.101 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:53 +0000 (0:00:00.056) 0:01:33.389 ********** 2021-03-12T02:22:53.119 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-12T02:22:53.121 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:53.152 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:53 +0000 (0:00:00.051) 0:01:33.441 ********** 2021-03-12T02:22:53.377 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-12T02:22:53.379 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:53.402 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:53 +0000 (0:00:00.249) 0:01:33.690 ********** 2021-03-12T02:22:53.512 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:53.513 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-03-12T02:22:53.515 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-03-12 02:20:20 UTC", "ActiveEnterTimestampMonotonic": "16777492", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target time-sync.target sysinit.target network.target var-run.mount systemd-tmpfiles-setup.service basic.target systemd-journald.socket local-fs.target system.slice -.mount nss-lookup.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-03-12 02:20:20 UTC", "AssertTimestampMonotonic": "16776455", "Before": "multi-user.target plymouth-quit.service getty@tty1.service xdm.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-03-12 02:20:20 UTC", "ConditionTimestampMonotonic": "16776455", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-03-12T02:22:53.517 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1185", "ExecMainStartTimestamp": "Fri 2021-03-12 02:20:20 UTC", "ExecMainStartTimestampMonotonic": "16777457", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-03-12 02:20:20 UTC", "InactiveExitTimestampMonotonic": "16777492", "InvocationID": "c9e16dd70cc54b07b5b8d49067b37f65", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2021-03-12T02:22:53.518 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1185", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2021-03-12T02:22:53.520 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-03-12 02:20:20 UTC", "StateChangeTimestampMonotonic": "16777492", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-03-12 02:20:20 UTC", "WatchdogTimestampMonotonic": "16777491", "WatchdogUSec": "0"}} 2021-03-12T02:22:53.540 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:53 +0000 (0:00:00.137) 0:01:33.828 ********** 2021-03-12T02:22:53.574 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:53 +0000 (0:00:00.034) 0:01:33.862 ********** 2021-03-12T02:22:53.641 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-03-12T02:22:53.643 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:53.668 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:53 +0000 (0:00:00.094) 0:01:33.957 ********** 2021-03-12T02:22:53.699 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-03-12T02:22:53.701 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:53.732 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:53 +0000 (0:00:00.063) 0:01:34.020 ********** 2021-03-12T02:22:53.783 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:53.808 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:53 +0000 (0:00:00.076) 0:01:34.096 ********** 2021-03-12T02:22:53.809 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:53.882 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-03-12T02:22:53.886 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:53.926 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:53 +0000 (0:00:00.118) 0:01:34.214 ********** 2021-03-12T02:22:53.934 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-03-12T02:22:53.936 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:53.971 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:53 +0000 (0:00:00.044) 0:01:34.259 ********** 2021-03-12T02:22:54.119 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:54.157 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:54 +0000 (0:00:00.186) 0:01:34.445 ********** 2021-03-12T02:22:54.182 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-03-12T02:22:54.184 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:54.223 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:54 +0000 (0:00:00.065) 0:01:34.511 ********** 2021-03-12T02:22:54.225 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:54.286 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:54.329 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:54 +0000 (0:00:00.105) 0:01:34.617 ********** 2021-03-12T02:22:54.333 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:54.374 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:54.376 INFO:teuthology.task.ansible.out:TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:54.409 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:54 +0000 (0:00:00.079) 0:01:34.697 ********** 2021-03-12T02:22:54.409 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:54.500 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:54.501 INFO:teuthology.task.ansible.out:TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-03-12T02:22:54.502 INFO:teuthology.task.ansible.out: skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:54.527 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:54 +0000 (0:00:00.118) 0:01:34.815 ********** 2021-03-12T02:22:54.528 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:54.540 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-03-12T02:22:54.542 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:54.585 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:54 +0000 (0:00:00.058) 0:01:34.874 ********** 2021-03-12T02:22:54.644 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-03-12T02:22:54.646 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:54.669 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:54 +0000 (0:00:00.084) 0:01:34.958 ********** 2021-03-12T02:22:54.731 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-03-12T02:22:54.733 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:54.760 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:54 +0000 (0:00:00.090) 0:01:35.049 ********** 2021-03-12T02:22:54.769 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:54.863 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:54.890 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:54 +0000 (0:00:00.129) 0:01:35.178 ********** 2021-03-12T02:22:54.998 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:55.024 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:55 +0000 (0:00:00.134) 0:01:35.312 ********** 2021-03-12T02:22:55.055 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005154", "end": "2021-03-12 02:22:55.496383", "rc": 0, "start": "2021-03-12 02:22:55.491229", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:22:55.112 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:55 +0000 (0:00:00.087) 0:01:35.400 ********** 2021-03-12T02:22:55.151 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-03-12T02:22:55.153 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:55.183 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:55 +0000 (0:00:00.065) 0:01:35.471 ********** 2021-03-12T02:22:55.255 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-03-12T02:22:55.293 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:55 +0000 (0:00:00.109) 0:01:35.581 ********** 2021-03-12T02:22:55.314 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-03-12T02:22:55.316 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:55.356 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:55 +0000 (0:00:00.063) 0:01:35.644 ********** 2021-03-12T02:22:55.543 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-03-12T02:22:55.545 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-03-12T02:22:55.571 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:55 +0000 (0:00:00.215) 0:01:35.859 ********** 2021-03-12T02:22:55.740 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-03-12T02:22:55.742 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-03-12T02:22:55.774 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:55 +0000 (0:00:00.200) 0:01:36.060 ********** 2021-03-12T02:22:55.876 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-03-12T02:22:55.879 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-03-12T02:22:55.938 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:55 +0000 (0:00:00.166) 0:01:36.226 ********** 2021-03-12T02:22:55.941 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:55.950 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-03-12T02:22:55.952 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-03-12T02:22:55.986 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:55 +0000 (0:00:00.048) 0:01:36.275 ********** 2021-03-12T02:22:56.035 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-03-12T02:22:56.037 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:56.068 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:56 +0000 (0:00:00.080) 0:01:36.355 ********** 2021-03-12T02:22:56.095 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-03-12T02:22:56.097 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-12T02:22:56.121 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:56 +0000 (0:00:00.053) 0:01:36.409 ********** 2021-03-12T02:22:56.269 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-03-12T02:22:56.271 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-12T02:22:56.307 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:56 +0000 (0:00:00.186) 0:01:36.595 ********** 2021-03-12T02:22:56.517 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-03-12T02:22:56.519 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-12T02:22:56.552 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:56 +0000 (0:00:00.244) 0:01:36.840 ********** 2021-03-12T02:22:56.652 INFO:teuthology.task.ansible.out: 2021-03-12T02:22:56.654 INFO:teuthology.task.ansible.out:TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:56.682 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:56 +0000 (0:00:00.130) 0:01:36.970 ********** 2021-03-12T02:22:56.804 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:22:56.825 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:56 +0000 (0:00:00.143) 0:01:37.113 ********** 2021-03-12T02:22:57.102 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi114.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-03-12T02:22:57.134 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:57 +0000 (0:00:00.309) 0:01:37.423 ********** 2021-03-12T02:22:57.781 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-12T02:22:57.783 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-03-12T02:22:57.856 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-12T02:22:57.858 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-03-12T02:22:57.889 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:57 +0000 (0:00:00.754) 0:01:38.177 ********** 2021-03-12T02:22:58.054 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi114.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi114"}, "changed": false} 2021-03-12T02:22:58.079 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:58 +0000 (0:00:00.189) 0:01:38.367 ********** 2021-03-12T02:22:58.868 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-12T02:22:58.871 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-03-12T02:22:58.911 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:58 +0000 (0:00:00.831) 0:01:39.199 ********** 2021-03-12T02:22:58.979 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ ok: [smithi114.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi114", "ansible_hostname": "smithi114", "ansible_nodename": "smithi114"}, "changed": false, "name": "smithi114"} 2021-03-12T02:22:59.005 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:59 +0000 (0:00:00.094) 0:01:39.294 ********** 2021-03-12T02:22:59.453 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-03-12T02:22:59.454 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-03-12T02:22:59.482 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:59 +0000 (0:00:00.476) 0:01:39.770 ********** 2021-03-12T02:22:59.928 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-03-12T02:22:59.930 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"cache_update_time": 1615515667, "cache_updated": false, "changed": false} 2021-03-12T02:22:59.953 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:59 +0000 (0:00:00.471) 0:01:40.241 ********** 2021-03-12T02:22:59.961 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-03-12T02:22:59.962 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-03-12T02:22:59.999 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:22:59 +0000 (0:00:00.045) 0:01:40.287 ********** 2021-03-12T02:23:00.076 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-03-12T02:23:00.108 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:00 +0000 (0:00:00.109) 0:01:40.396 ********** 2021-03-12T02:23:00.436 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-03-12T02:23:00.437 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.037130", "end": "2021-03-12 02:23:00.880298", "rc": 0, "start": "2021-03-12 02:23:00.843168", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:00.462 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:00 +0000 (0:00:00.351) 0:01:40.748 ********** 2021-03-12T02:23:00.499 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** changed: [smithi114.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-12T02:23:00.531 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:00 +0000 (0:00:00.071) 0:01:40.819 ********** 2021-03-12T02:23:00.688 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-03-12T02:23:00.690 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:00.716 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:00 +0000 (0:00:00.184) 0:01:41.004 ********** 2021-03-12T02:23:00.814 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-12T02:23:00.816 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:00.840 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:00 +0000 (0:00:00.124) 0:01:41.128 ********** 2021-03-12T02:23:00.921 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:00.948 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:00 +0000 (0:00:00.108) 0:01:41.236 ********** 2021-03-12T02:23:01.017 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-12T02:23:01.019 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:01.040 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:01 +0000 (0:00:00.092) 0:01:41.329 ********** 2021-03-12T02:23:01.120 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-12T02:23:01.122 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:01.150 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:01 +0000 (0:00:00.109) 0:01:41.438 ********** 2021-03-12T02:23:01.243 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-12T02:23:01.246 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:01.272 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:01 +0000 (0:00:00.121) 0:01:41.560 ********** 2021-03-12T02:23:01.365 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-12T02:23:01.367 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:01.394 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:01 +0000 (0:00:00.122) 0:01:41.682 ********** 2021-03-12T02:23:01.484 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-03-12T02:23:01.486 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:01.508 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:01 +0000 (0:00:00.114) 0:01:41.796 ********** 2021-03-12T02:23:01.612 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-03-12T02:23:01.639 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:01 +0000 (0:00:00.130) 0:01:41.927 ********** 2021-03-12T02:23:01.735 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-03-12T02:23:01.760 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:01 +0000 (0:00:00.121) 0:01:42.048 ********** 2021-03-12T02:23:01.844 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:01.846 INFO:teuthology.task.ansible.out:TASK [testnode : Enable copr repos] ******************************************** 2021-03-12T02:23:01.869 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:01 +0000 (0:00:00.108) 0:01:42.157 ********** 2021-03-12T02:23:01.870 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:01.961 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:01.985 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:01 +0000 (0:00:00.116) 0:01:42.273 ********** 2021-03-12T02:23:02.089 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-03-12T02:23:02.091 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:02.101 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:02.125 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:02 +0000 (0:00:00.140) 0:01:42.414 ********** 2021-03-12T02:23:02.203 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-03-12T02:23:02.204 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:02.236 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:02 +0000 (0:00:00.111) 0:01:42.525 ********** 2021-03-12T02:23:02.345 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:02.394 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:02 +0000 (0:00:00.157) 0:01:42.682 ********** 2021-03-12T02:23:02.479 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-03-12T02:23:02.481 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:02.504 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:02 +0000 (0:00:00.110) 0:01:42.792 ********** 2021-03-12T02:23:02.596 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:02.619 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:02 +0000 (0:00:00.114) 0:01:42.907 ********** 2021-03-12T02:23:02.712 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-03-12T02:23:02.714 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:02.752 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:02 +0000 (0:00:00.133) 0:01:43.040 ********** 2021-03-12T02:23:02.831 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-03-12T02:23:02.832 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:02.854 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:02 +0000 (0:00:00.102) 0:01:43.143 ********** 2021-03-12T02:23:02.956 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-12T02:23:02.959 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:02.980 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:02 +0000 (0:00:00.125) 0:01:43.268 ********** 2021-03-12T02:23:02.981 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:03.086 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-12T02:23:03.088 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:03.127 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:03 +0000 (0:00:00.147) 0:01:43.415 ********** 2021-03-12T02:23:03.332 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:03.364 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:03 +0000 (0:00:00.235) 0:01:43.651 ********** 2021-03-12T02:23:03.534 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-03-12T02:23:03.536 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"cache_update_time": 1615515667, "cache_updated": false, "changed": false} 2021-03-12T02:23:03.565 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:03 +0000 (0:00:00.202) 0:01:43.853 ********** 2021-03-12T02:23:03.569 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:03.616 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-03-12T02:23:03.618 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:03.651 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:03 +0000 (0:00:00.085) 0:01:43.939 ********** 2021-03-12T02:23:03.853 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-12T02:23:03.855 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:03.878 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:03 +0000 (0:00:00.227) 0:01:44.166 ********** 2021-03-12T02:23:04.027 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-12T02:23:04.029 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:04.059 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:04 +0000 (0:00:00.181) 0:01:44.347 ********** 2021-03-12T02:23:04.208 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-03-12T02:23:04.210 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:04.252 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:04 +0000 (0:00:00.193) 0:01:44.541 ********** 2021-03-12T02:23:04.465 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-03-12T02:23:04.470 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:04.490 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:04 +0000 (0:00:00.237) 0:01:44.778 ********** 2021-03-12T02:23:04.493 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:04.622 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:04.643 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:04 +0000 (0:00:00.153) 0:01:44.932 ********** 2021-03-12T02:23:04.759 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-03-12T02:23:04.761 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:04.783 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:04 +0000 (0:00:00.139) 0:01:45.071 ********** 2021-03-12T02:23:04.865 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-03-12T02:23:04.868 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:04.901 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:04 +0000 (0:00:00.117) 0:01:45.189 ********** 2021-03-12T02:23:04.903 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:04.983 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-03-12T02:23:04.984 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:05.006 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:05 +0000 (0:00:00.104) 0:01:45.294 ********** 2021-03-12T02:23:05.118 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:05.144 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:05 +0000 (0:00:00.138) 0:01:45.433 ********** 2021-03-12T02:23:05.228 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-03-12T02:23:05.229 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:05.275 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:05 +0000 (0:00:00.130) 0:01:45.563 ********** 2021-03-12T02:23:05.483 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005161", "end": "2021-03-12 02:23:05.330995", "rc": 0, "start": "2021-03-12 02:23:05.325834", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:05.508 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:05 +0000 (0:00:00.233) 0:01:45.797 ********** 2021-03-12T02:23:05.510 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:05.581 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-03-12T02:23:05.607 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:05 +0000 (0:00:00.098) 0:01:45.895 ********** 2021-03-12T02:23:06.016 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-03-12T02:23:06.018 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-03-12T02:23:06.043 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:06 +0000 (0:00:00.435) 0:01:46.331 ********** 2021-03-12T02:23:06.125 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-03-12T02:23:06.128 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:06.150 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:06 +0000 (0:00:00.107) 0:01:46.438 ********** 2021-03-12T02:23:06.152 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:07.292 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-12T02:23:07.294 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-03-12T02:23:08.329 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-03-12T02:23:08.354 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:08 +0000 (0:00:02.203) 0:01:48.642 ********** 2021-03-12T02:23:09.379 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-03-12T02:23:09.384 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"cache_update_time": 1615515672, "cache_updated": false, "changed": false} 2021-03-12T02:23:09.417 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:09 +0000 (0:00:01.063) 0:01:49.706 ********** 2021-03-12T02:23:09.501 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-03-12T02:23:09.549 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:09 +0000 (0:00:00.131) 0:01:49.837 ********** 2021-03-12T02:23:09.551 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:09.825 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-03-12T02:23:09.827 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.030097", "end": "2021-03-12 02:23:09.662513", "rc": 0, "start": "2021-03-12 02:23:09.632416", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:09.849 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:09 +0000 (0:00:00.300) 0:01:50.137 ********** 2021-03-12T02:23:12.799 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-03-12T02:23:12.801 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"cache_update_time": 1615515672, "cache_updated": false, "changed": false} 2021-03-12T02:23:12.854 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:12 +0000 (0:00:03.004) 0:01:53.142 ********** 2021-03-12T02:23:13.986 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-03-12T02:23:13.988 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 18 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102064 files and directorie 2021-03-12T02:23:13.990 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 18 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-03-12T02:23:13.991 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102064 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-12T02:23:14.012 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:14 +0000 (0:00:01.158) 0:01:54.301 ********** 2021-03-12T02:23:15.019 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-12T02:23:15.021 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"cache_update_time": 1615515667, "cache_updated": false, "changed": false} 2021-03-12T02:23:15.045 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:15 +0000 (0:00:01.032) 0:01:55.333 ********** 2021-03-12T02:23:15.126 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-03-12T02:23:15.128 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:15.151 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:15 +0000 (0:00:00.106) 0:01:55.439 ********** 2021-03-12T02:23:23.377 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:23.379 INFO:teuthology.task.ansible.out:TASK [testnode : Ensure packages are not present.] ***************************** changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 18 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 2021-03-12T02:23:23.381 INFO:teuthology.task.ansible.out:95%\r(Reading database ... 100%\r(Reading database ... 102064 files and directories currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 18 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading d 2021-03-12T02:23:23.382 INFO:teuthology.task.ansible.out:atabase ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102064 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-12T02:23:23.413 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:23 +0000 (0:00:08.261) 0:02:03.701 ********** 2021-03-12T02:23:24.367 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-12T02:23:24.369 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"cache_update_time": 1615515672, "cache_updated": false, "changed": false} 2021-03-12T02:23:24.398 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:24 +0000 (0:00:00.980) 0:02:04.682 ********** 2021-03-12T02:23:24.487 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-03-12T02:23:24.489 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:24.512 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:24 +0000 (0:00:00.117) 0:02:04.800 ********** 2021-03-12T02:23:25.539 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-12T02:23:25.540 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"cache_update_time": 1615515667, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk\n gitweb git-cvs git-mediawiki git-svn samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\nThe following packages will be upgraded:\n git\n1 upgraded, 6 newly installed, 0 to remove and 17 not upgraded.\nNeed to get 14.3 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.8 [3916 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]\nGet:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]\nFetched 14.3 MB in 1s (12.0 MB/s)\nSelecting previously unselected package librados2.\r\n(Reading database .. 2021-03-12T02:23:25.541 INFO:teuthology.task.ansible.out:. \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101922 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nPreparing to unpack .../4-git_1%3a2.17.1-1ubuntu0.8_amd64.deb ...\r\nUnpacking git (1:2.17.1-1ubuntu0.8) over (1:2.17.1-1ubuntu0.7) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../5-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../6-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up git (1:2.17.1-1ubuntu0.8) ...\r\nSetti 2021-03-12T02:23:25.542 INFO:teuthology.task.ansible.out:ng up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk", " gitweb git-cvs git-mediawiki git-svn samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "The following packages will be upgraded:", " git", "1 upgraded, 6 newly installed, 0 to remove and 17 not upgraded.", "Need to get 14.3 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.8 [3916 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]", "Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.1 2021-03-12T02:23:25.544 INFO:teuthology.task.ansible.out:1+dfsg-1ubuntu7.36 [870 kB]", "Fetched 14.3 MB in 1s (12.0 MB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101922 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Preparing to unpack .../4-git_1%3a2.17.1-1ubuntu0.8_amd64.deb ...", "Unpacking git (1:2.17.1-1ubuntu0.8) over (1:2.17.1-1ubuntu0.7) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../5-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../6-qemu-utils_1%3a2.11+dfsg-1ub 2021-03-12T02:23:25.545 INFO:teuthology.task.ansible.out:untu7.36_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up git (1:2.17.1-1ubuntu0.8) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-12T02:23:25.567 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:25 +0000 (0:00:01.054) 0:02:05.855 ********** 2021-03-12T02:23:26.543 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-03-12T02:23:26.545 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"cache_update_time": 1615515667, "cache_updated": false, "changed": false} 2021-03-12T02:23:26.569 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:26 +0000 (0:00:01.002) 0:02:06.857 ********** 2021-03-12T02:23:27.570 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-03-12T02:23:27.572 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"cache_update_time": 1615515667, "cache_updated": false, "changed": false} 2021-03-12T02:23:27.597 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:27 +0000 (0:00:01.027) 0:02:07.885 ********** 2021-03-12T02:23:27.816 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-12T02:23:27.818 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-03-12T02:23:27.841 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:27 +0000 (0:00:00.244) 0:02:08.129 ********** 2021-03-12T02:23:27.937 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-03-12T02:23:27.939 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:27.964 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:27 +0000 (0:00:00.123) 0:02:08.253 ********** 2021-03-12T02:23:28.255 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-12T02:23:28.256 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-03-12T02:23:28.281 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:28 +0000 (0:00:00.316) 0:02:08.569 ********** 2021-03-12T02:23:28.796 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-03-12T02:23:28.820 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:28 +0000 (0:00:00.538) 0:02:09.108 ********** 2021-03-12T02:23:29.089 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-03-12T02:23:29.091 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-12T02:23:29.114 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:29 +0000 (0:00:00.294) 0:02:09.403 ********** 2021-03-12T02:23:29.316 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-03-12T02:23:29.318 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-12T02:23:29.345 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:29 +0000 (0:00:00.230) 0:02:09.633 ********** 2021-03-12T02:23:29.748 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-03-12T02:23:29.749 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Fri 2021-03-12 02:20:24 UTC", "ActiveEnterTimestampMonotonic": "17728356", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target system.slice nss-lookup.target basic.target systemd-tmpfiles-setup.service systemd-journald.socket sysinit.target network.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-03-12 02:20:24 UTC", "AssertTimestampMonotonic": "17345423", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-03-12 02:20:24 UTC", "ConditionTimestampMonotonic": "17345423", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-03-12T02:23:29.751 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1256", "ExecMainStartTimestamp": "Fri 2021-03-12 02:20:24 UTC", "ExecMainStartTimestampMonotonic": "17728346", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-03-12 02:20:24 UTC", "InactiveExitTimestampMonotonic": "17346081", "InvocationID": "b09787638c664da4b1498d43cd104bce", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-03-12T02:23:29.752 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1256", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-03-12T02:23:29.754 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-03-12 02:20:24 UTC", "StateChangeTimestampMonotonic": "17728356", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-03-12 02:20:24 UTC", "WatchdogTimestampMonotonic": "17728355", "WatchdogUSec": "0"}} 2021-03-12T02:23:29.776 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:29 +0000 (0:00:00.429) 0:02:10.063 ********** 2021-03-12T02:23:29.868 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-12T02:23:29.869 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:29.897 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:29 +0000 (0:00:00.122) 0:02:10.185 ********** 2021-03-12T02:23:29.966 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:29.990 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:29 +0000 (0:00:00.092) 0:02:10.278 ********** 2021-03-12T02:23:30.060 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-12T02:23:30.062 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:30.083 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:30 +0000 (0:00:00.093) 0:02:10.372 ********** 2021-03-12T02:23:30.164 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-12T02:23:30.168 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:30.188 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:30 +0000 (0:00:00.104) 0:02:10.476 ********** 2021-03-12T02:23:30.257 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-12T02:23:30.259 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:30.287 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:30 +0000 (0:00:00.099) 0:02:10.575 ********** 2021-03-12T02:23:30.358 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-12T02:23:30.360 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:30.382 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:30 +0000 (0:00:00.094) 0:02:10.670 ********** 2021-03-12T02:23:30.464 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-12T02:23:30.466 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:30.488 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:30 +0000 (0:00:00.105) 0:02:10.776 ********** 2021-03-12T02:23:30.570 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-12T02:23:30.572 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:30.595 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:30 +0000 (0:00:00.106) 0:02:10.883 ********** 2021-03-12T02:23:30.668 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-12T02:23:30.670 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:30.693 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:30 +0000 (0:00:00.098) 0:02:10.982 ********** 2021-03-12T02:23:30.773 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:30.796 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:30 +0000 (0:00:00.102) 0:02:11.084 ********** 2021-03-12T02:23:30.875 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-12T02:23:30.877 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:30.901 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:30 +0000 (0:00:00.104) 0:02:11.189 ********** 2021-03-12T02:23:30.973 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-12T02:23:30.975 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:30.997 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:30 +0000 (0:00:00.096) 0:02:11.286 ********** 2021-03-12T02:23:31.077 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-12T02:23:31.079 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:31.104 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:31 +0000 (0:00:00.106) 0:02:11.393 ********** 2021-03-12T02:23:31.185 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-12T02:23:31.187 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:31.208 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:31 +0000 (0:00:00.103) 0:02:11.496 ********** 2021-03-12T02:23:31.277 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:31.300 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:31 +0000 (0:00:00.092) 0:02:11.589 ********** 2021-03-12T02:23:31.381 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-12T02:23:31.383 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:31.420 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:31 +0000 (0:00:00.119) 0:02:11.709 ********** 2021-03-12T02:23:31.522 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-12T02:23:31.524 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:31.547 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:31 +0000 (0:00:00.126) 0:02:11.835 ********** 2021-03-12T02:23:31.638 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-12T02:23:31.640 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:31.665 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:31 +0000 (0:00:00.118) 0:02:11.953 ********** 2021-03-12T02:23:31.747 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-12T02:23:31.749 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:31.771 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:31 +0000 (0:00:00.106) 0:02:12.060 ********** 2021-03-12T02:23:31.846 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-12T02:23:31.848 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:31.869 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:31 +0000 (0:00:00.098) 0:02:12.158 ********** 2021-03-12T02:23:31.950 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-12T02:23:31.952 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:31.973 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:31 +0000 (0:00:00.103) 0:02:12.262 ********** 2021-03-12T02:23:32.056 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:32.082 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:32 +0000 (0:00:00.108) 0:02:12.370 ********** 2021-03-12T02:23:32.083 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:32.165 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:32.203 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:32 +0000 (0:00:00.120) 0:02:12.491 ********** 2021-03-12T02:23:32.316 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:32.342 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:32 +0000 (0:00:00.139) 0:02:12.630 ********** 2021-03-12T02:23:32.423 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-12T02:23:32.424 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:32.455 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:32 +0000 (0:00:00.112) 0:02:12.743 ********** 2021-03-12T02:23:32.529 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-03-12T02:23:32.530 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:32.558 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:32 +0000 (0:00:00.103) 0:02:12.847 ********** 2021-03-12T02:23:32.853 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-03-12T02:23:32.855 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-03-12T02:23:32.879 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:32 +0000 (0:00:00.320) 0:02:13.167 ********** 2021-03-12T02:23:33.381 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-12T02:23:33.383 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-03-12T02:23:33.408 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:33 +0000 (0:00:00.529) 0:02:13.696 ********** 2021-03-12T02:23:33.412 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:33.498 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:33.499 INFO:teuthology.task.ansible.out:TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:33.525 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:33 +0000 (0:00:00.116) 0:02:13.813 ********** 2021-03-12T02:23:34.007 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-03-12T02:23:34.009 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-03-12T02:23:34.035 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:34 +0000 (0:00:00.508) 0:02:14.321 ********** 2021-03-12T02:23:34.584 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-03-12T02:23:34.622 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:34 +0000 (0:00:00.587) 0:02:14.909 ********** 2021-03-12T02:23:34.856 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-12T02:23:34.858 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"cache_update_time": 1615515672, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk\n gitweb git-cvs git-mediawiki git-svn samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\nThe following packages will be upgraded:\n git\n1 upgraded, 6 newly installed, 0 to remove and 17 not upgraded.\nNeed to get 14.3 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.8 [3916 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]\nGet:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]\nFetched 14.3 MB in 1s (11.7 MB/s)\nSelecting previously unselected package librados2.\r\n(Reading database .. 2021-03-12T02:23:34.861 INFO:teuthology.task.ansible.out:. \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101922 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nPreparing to unpack .../4-git_1%3a2.17.1-1ubuntu0.8_amd64.deb ...\r\nUnpacking git (1:2.17.1-1ubuntu0.8) over (1:2.17.1-1ubuntu0.7) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../5-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../6-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up git (1:2.17.1-1ubuntu0.8) ...\r\nSetti 2021-03-12T02:23:34.862 INFO:teuthology.task.ansible.out:ng up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk", " gitweb git-cvs git-mediawiki git-svn samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "The following packages will be upgraded:", " git", "1 upgraded, 6 newly installed, 0 to remove and 17 not upgraded.", "Need to get 14.3 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.8 [3916 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]", "Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.1 2021-03-12T02:23:34.865 INFO:teuthology.task.ansible.out:1+dfsg-1ubuntu7.36 [870 kB]", "Fetched 14.3 MB in 1s (11.7 MB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101922 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Preparing to unpack .../4-git_1%3a2.17.1-1ubuntu0.8_amd64.deb ...", "Unpacking git (1:2.17.1-1ubuntu0.8) over (1:2.17.1-1ubuntu0.7) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../5-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../6-qemu-utils_1%3a2.11+dfsg-1ub 2021-03-12T02:23:34.867 INFO:teuthology.task.ansible.out:untu7.36_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up git (1:2.17.1-1ubuntu0.8) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-12T02:23:34.892 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:34 +0000 (0:00:00.270) 0:02:15.180 ********** 2021-03-12T02:23:35.049 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-03-12T02:23:35.073 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:35 +0000 (0:00:00.181) 0:02:15.361 ********** 2021-03-12T02:23:35.269 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-03-12T02:23:35.271 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-12T02:23:35.294 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:35 +0000 (0:00:00.220) 0:02:15.582 ********** 2021-03-12T02:23:35.374 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-03-12T02:23:35.376 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:35.398 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:35 +0000 (0:00:00.104) 0:02:15.687 ********** 2021-03-12T02:23:35.512 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-03-12T02:23:35.514 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:35.539 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:35 +0000 (0:00:00.140) 0:02:15.827 ********** 2021-03-12T02:23:35.609 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-03-12T02:23:35.611 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:35.645 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:35 +0000 (0:00:00.105) 0:02:15.933 ********** 2021-03-12T02:23:35.736 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-03-12T02:23:35.738 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:35.763 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:35 +0000 (0:00:00.118) 0:02:16.051 ********** 2021-03-12T02:23:35.835 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-12T02:23:35.838 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:35.862 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:35 +0000 (0:00:00.098) 0:02:16.150 ********** 2021-03-12T02:23:35.870 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-03-12T02:23:35.871 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"cache_update_time": 1615515672, "cache_updated": false, "changed": false} 2021-03-12T02:23:35.897 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:35 +0000 (0:00:00.035) 0:02:16.186 ********** 2021-03-12T02:23:35.936 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-12T02:23:35.938 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-12T02:23:35.962 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:35 +0000 (0:00:00.063) 0:02:16.249 ********** 2021-03-12T02:23:36.073 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-12T02:23:36.075 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:36.085 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:36.096 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:36.107 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:36.129 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-03-12T02:23:36.131 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:36.154 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:36.156 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:36.166 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:36.168 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:36.178 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:36.181 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:36.201 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:36.233 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:36 +0000 (0:00:00.272) 0:02:16.522 ********** 2021-03-12T02:23:36.815 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-03-12T02:23:36.817 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"cache_update_time": 1615515672, "cache_updated": false, "changed": false} 2021-03-12T02:23:36.842 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:36 +0000 (0:00:00.608) 0:02:17.130 ********** 2021-03-12T02:23:36.864 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-03-12T02:23:36.866 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.053045", "end": "2021-03-12 02:23:36.820063", "rc": 0, "start": "2021-03-12 02:23:36.767018", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock"]} 2021-03-12T02:23:36.928 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:36 +0000 (0:00:00.086) 0:02:17.216 ********** 2021-03-12T02:23:37.103 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-03-12T02:23:37.105 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:37.146 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:37 +0000 (0:00:00.210) 0:02:17.427 ********** 2021-03-12T02:23:37.219 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-12T02:23:37.221 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-03-12T02:23:37.279 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:37 +0000 (0:00:00.140) 0:02:17.567 ********** 2021-03-12T02:23:37.291 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:37.299 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-03-12T02:23:37.300 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:37.377 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:37 +0000 (0:00:00.097) 0:02:17.665 ********** 2021-03-12T02:23:37.449 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-03-12T02:23:37.451 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:37.485 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:37 +0000 (0:00:00.108) 0:02:17.773 ********** 2021-03-12T02:23:37.526 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-03-12T02:23:37.528 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:37.565 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:37 +0000 (0:00:00.079) 0:02:17.853 ********** 2021-03-12T02:23:37.745 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-12T02:23:37.747 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-03-12T02:23:37.794 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:37 +0000 (0:00:00.228) 0:02:18.082 ********** 2021-03-12T02:23:38.233 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-03-12T02:23:38.235 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-03-12T02:23:38.258 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:38 +0000 (0:00:00.464) 0:02:18.547 ********** 2021-03-12T02:23:38.524 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ ok: [smithi114.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-12T02:23:38.550 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:38 +0000 (0:00:00.291) 0:02:18.838 ********** 2021-03-12T02:23:38.568 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-03-12T02:23:38.570 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"cache_update_time": 1615515667, "cache_updated": false, "changed": false} 2021-03-12T02:23:38.602 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:38 +0000 (0:00:00.050) 0:02:18.888 ********** 2021-03-12T02:23:38.681 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-12T02:23:38.682 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:38.705 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:38 +0000 (0:00:00.105) 0:02:18.994 ********** 2021-03-12T02:23:38.767 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-03-12T02:23:38.769 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-12T02:23:38.798 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:38 +0000 (0:00:00.092) 0:02:19.086 ********** 2021-03-12T02:23:38.806 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-12T02:23:38.806 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'block_used': 13643750, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451358, 'size_available': 927544762368, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451358, "block_size": 4096, "block_total": 240095108, "block_used": 13643750, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544762368, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}} 2021-03-12T02:23:38.840 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:38 +0000 (0:00:00.042) 0:02:19.129 ********** 2021-03-12T02:23:39.027 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-12T02:23:39.029 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.006885", "end": "2021-03-12 02:23:39.499925", "rc": 0, "start": "2021-03-12 02:23:39.493040", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:39.051 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:39 +0000 (0:00:00.211) 0:02:19.340 ********** 2021-03-12T02:23:39.146 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:39.148 INFO:teuthology.task.ansible.out:TASK [testnode : Stop apache2] ************************************************* 2021-03-12T02:23:39.150 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:39.153 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Fri 2021-03-12 02:20:21 UTC", "ActiveEnterTimestampMonotonic": "17364969", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service -.mount sysinit.target nss-lookup.target systemd-journald.socket remote-fs.target basic.target network.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-03-12 02:20:20 UTC", "AssertTimestampMonotonic": "16774544", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-03-12 02:20:20 UTC", "ConditionTimestampMonotonic": "16774544", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-03-12T02:23:39.154 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1275", "ExecMainStartTimestamp": "Fri 2021-03-12 02:20:21 UTC", "ExecMainStartTimestampMonotonic": "17364957", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-03-12 02:20:20 UTC", "InactiveExitTimestampMonotonic": "16775448", "InvocationID": "d542452eb85041c881f2a7fa51c1b98e", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-03-12T02:23:39.156 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1275", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-03-12T02:23:39.157 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-03-12 02:20:21 UTC", "StateChangeTimestampMonotonic": "17364969", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-03-12 02:20:21 UTC", "WatchdogTimestampMonotonic": "17364968", "WatchdogUSec": "0"}} TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi077.front.sepia.ceph.com] => (item={'block_used': 13643750, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451358, 'size_available': 927544762368, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451358, "block_size": 4096, "block_total": 240095108, "block_used": 13643750, "device": "/dev/sda1", "fstype": "ext4", "inode 2021-03-12T02:23:39.159 INFO:teuthology.task.ansible.out:_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544762368, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:39.178 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:39 +0000 (0:00:00.126) 0:02:19.467 ********** 2021-03-12T02:23:39.206 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:39 +0000 (0:00:00.027) 0:02:19.494 ********** 2021-03-12T02:23:39.260 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-03-12T02:23:39.285 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:39 +0000 (0:00:00.079) 0:02:19.573 ********** 2021-03-12T02:23:39.303 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-12T02:23:39.305 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:39.330 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:39 +0000 (0:00:00.045) 0:02:19.618 ********** 2021-03-12T02:23:39.410 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-12T02:23:39.411 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:39.438 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:39 +0000 (0:00:00.107) 0:02:19.726 ********** 2021-03-12T02:23:39.447 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-12T02:23:39.449 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:39.526 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-12T02:23:39.528 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:39.557 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:39 +0000 (0:00:00.119) 0:02:19.846 ********** 2021-03-12T02:23:39.640 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:39.667 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:39 +0000 (0:00:00.109) 0:02:19.955 ********** 2021-03-12T02:23:39.758 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-12T02:23:39.760 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:39.781 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:39 +0000 (0:00:00.113) 0:02:20.069 ********** 2021-03-12T02:23:39.850 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-12T02:23:39.852 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:39.875 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:39 +0000 (0:00:00.093) 0:02:20.163 ********** 2021-03-12T02:23:39.946 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:39.969 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:39 +0000 (0:00:00.094) 0:02:20.257 ********** 2021-03-12T02:23:40.038 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-12T02:23:40.040 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:40.063 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:40 +0000 (0:00:00.094) 0:02:20.352 ********** 2021-03-12T02:23:40.144 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:40.168 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:40 +0000 (0:00:00.104) 0:02:20.456 ********** 2021-03-12T02:23:40.239 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-12T02:23:40.241 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:40.262 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:40 +0000 (0:00:00.094) 0:02:20.551 ********** 2021-03-12T02:23:40.343 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-12T02:23:40.346 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:40.370 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:40 +0000 (0:00:00.107) 0:02:20.659 ********** 2021-03-12T02:23:40.460 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:40.464 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:40.484 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:40 +0000 (0:00:00.113) 0:02:20.772 ********** 2021-03-12T02:23:40.563 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-12T02:23:40.565 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:40.589 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:40 +0000 (0:00:00.105) 0:02:20.877 ********** 2021-03-12T02:23:40.681 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-12T02:23:40.683 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:40.704 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:40 +0000 (0:00:00.115) 0:02:20.993 ********** 2021-03-12T02:23:40.785 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-12T02:23:40.786 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:40.817 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:40 +0000 (0:00:00.112) 0:02:21.106 ********** 2021-03-12T02:23:40.888 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-12T02:23:40.890 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:40.912 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:40 +0000 (0:00:00.094) 0:02:21.201 ********** 2021-03-12T02:23:40.993 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:41.020 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:41 +0000 (0:00:00.107) 0:02:21.308 ********** 2021-03-12T02:23:41.069 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-12T02:23:41.071 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbgtoJNz3OKiDDfzY7Rf8aKDT4yOd6PhK6z'], 'uuids': ['f6537e64-26b5-4e25-b34f-1f57adffedbc']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.522014", "end": "2021-03-12 02:23:41.521735", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbgtoJNz3OKiDDfzY7Rf8aKDT4yOd6PhK6z"], "labels": [], "masters": [], "uuids": ["f6537e64-26b5-4e25-b34f-1f57adffedbc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:39.999721", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-12T02:23:41.071 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => 2021-03-12T02:23:41.073 INFO:teuthology.task.ansible.out:(item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:41.083 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:41.094 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:41.122 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:41 +0000 (0:00:00.101) 0:02:21.410 ********** 2021-03-12T02:23:41.204 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-12T02:23:41.205 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:41.229 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:41 +0000 (0:00:00.107) 0:02:21.518 ********** 2021-03-12T02:23:41.299 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-12T02:23:41.300 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:41.326 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:41 +0000 (0:00:00.097) 0:02:21.615 ********** 2021-03-12T02:23:41.396 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-12T02:23:41.399 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:41.421 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:41 +0000 (0:00:00.094) 0:02:21.709 ********** 2021-03-12T02:23:41.540 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:41.562 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:41 +0000 (0:00:00.141) 0:02:21.851 ********** 2021-03-12T02:23:41.641 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-12T02:23:41.643 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:41.666 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:41 +0000 (0:00:00.103) 0:02:21.954 ********** 2021-03-12T02:23:41.768 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:41.790 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:41 +0000 (0:00:00.124) 0:02:22.079 ********** 2021-03-12T02:23:41.871 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-12T02:23:41.873 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:41.894 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:41 +0000 (0:00:00.103) 0:02:22.182 ********** 2021-03-12T02:23:42.019 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-03-12T02:23:42.021 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:42.044 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:42 +0000 (0:00:00.150) 0:02:22.332 ********** 2021-03-12T02:23:42.334 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbgTqy1uH7IhmLis24f9lXYUn9LVSBMyFfX'], 'uuids': ['7e882b4c-578f-4f2d-9998-a2c0379877ef']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.111334", "end": "2021-03-12 02:23:42.784235", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbgTqy1uH7IhmLis24f9lXYUn9LVSBMyFfX"], "labels": [], "masters": [], "uuids": ["7e882b4c-578f-4f2d-9998-a2c0379877ef"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:41.672901", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk u 2021-03-12T02:23:42.339 INFO:teuthology.task.ansible.out:sing fdisk or", "other utilities."]} skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJQN', 'wwn-0x5000c500918d87b8'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJQN-part1', 'wwn-0x5000c500918d87b8-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918d87b8', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LJQN', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJQN", "wwn-0x5000c500918d87b8"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJQN-part1", "wwn-0x5000c500918d87b8-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LJQN", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": " 2021-03-12T02:23:42.340 INFO:teuthology.task.ansible.out:0x5000c500918d87b8"}}, "skip_reason": "Conditional result was False"} TASK [testnode : Remove /etc/ceph] ********************************************* ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-03-12T02:23:42.367 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:42 +0000 (0:00:00.322) 0:02:22.655 ********** 2021-03-12T02:23:42.369 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:42.863 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-12T02:23:42.864 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-03-12T02:23:42.896 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:42 +0000 (0:00:00.529) 0:02:23.184 ********** 2021-03-12T02:23:42.967 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-12T02:23:42.969 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:42.992 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:42 +0000 (0:00:00.096) 0:02:23.280 ********** 2021-03-12T02:23:43.432 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:43.433 INFO:teuthology.task.ansible.out:TASK [testnode : Upload weekly kernel-clean crontab.] ************************** ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-03-12T02:23:43.459 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:43 +0000 (0:00:00.466) 0:02:23.747 ********** 2021-03-12T02:23:43.644 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-12T02:23:43.645 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-YqYRpJ-x9wa-t3Y2-i9Lc-Buje-hdmp-RsNyuB', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400ZJ400BGN', 'nvme-nvme.8086-504846543632303430305a4a34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.160993", "end": "2021-03-12 02:23:44.096741", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-YqYRpJ-x9wa-t3Y2-i9Lc-Buje-hdmp-RsNyuB", "nvme-INTEL_SSDPEDMD400G4_PHFT620400ZJ400BGN", "nvme-nvme.8086-504846543632303430305a4a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:42.935748", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-03-12T02:23:43.647 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-12T02:23:43.656 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:43.658 INFO:teuthology.task.ansible.out: skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:43.903 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-03-12T02:23:43.905 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-03-12T02:23:43.927 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:43 +0000 (0:00:00.468) 0:02:24.215 ********** 2021-03-12T02:23:44.413 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-03-12T02:23:44.612 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} Friday 12 March 2021 02:23:44 +0000 (0:00:00.509) 0:02:24.725 ********** 2021-03-12T02:23:44.627 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** ok: [smithi114.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-12T02:23:44.651 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:44 +0000 (0:00:00.214) 0:02:24.940 ********** 2021-03-12T02:23:44.739 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-03-12T02:23:44.741 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:44.767 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:44 +0000 (0:00:00.115) 0:02:25.055 ********** 2021-03-12T02:23:44.860 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-03-12T02:23:44.862 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:44.884 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:44 +0000 (0:00:00.117) 0:02:25.172 ********** 2021-03-12T02:23:44.963 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-03-12T02:23:44.964 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:44.987 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:44 +0000 (0:00:00.102) 0:02:25.275 ********** 2021-03-12T02:23:44.996 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-12T02:23:44.997 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbg3OfBdAAd2kO1bfc9m1KicR04T5kSWM1D'], 'uuids': ['b34620f1-7a5b-4eb3-ab3c-c1c85fb239f8']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.153501", "end": "2021-03-12 02:23:45.425776", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbg3OfBdAAd2kO1bfc9m1KicR04T5kSWM1D"], "labels": [], "masters": [], "uuids": ["b34620f1-7a5b-4eb3-ab3c-c1c85fb239f8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:44.272275", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-12T02:23:45.000 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:45.002 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:45.004 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:45.080 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:45.104 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:45 +0000 (0:00:00.117) 0:02:25.392 ********** 2021-03-12T02:23:45.209 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-12T02:23:45.210 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:45.232 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:45 +0000 (0:00:00.127) 0:02:25.520 ********** 2021-03-12T02:23:45.326 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-12T02:23:45.328 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-12T02:23:45.351 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:45 +0000 (0:00:00.118) 0:02:25.639 ********** 2021-03-12T02:23:45.485 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-12T02:23:45.487 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:45.497 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:45.499 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:45.510 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:45.521 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-03-12T02:23:45.532 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:45.534 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:45.543 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:45.554 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:45.556 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:45.566 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:45.578 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:45.580 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:45.581 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-12T02:23:45.610 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:45 +0000 (0:00:00.259) 0:02:25.898 ********** 2021-03-12T02:23:45.947 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-03-12T02:23:45.949 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.005000", "end": "2021-03-12 02:23:45.914023", "rc": 0, "start": "2021-03-12 02:23:45.909023", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock"]} 2021-03-12T02:23:45.975 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:45 +0000 (0:00:00.364) 0:02:26.263 ********** 2021-03-12T02:23:46.086 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-03-12T02:23:46.088 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:46.115 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:46 +0000 (0:00:00.137) 0:02:26.401 ********** 2021-03-12T02:23:46.203 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-03-12T02:23:46.205 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:46.226 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:46 +0000 (0:00:00.113) 0:02:26.515 ********** 2021-03-12T02:23:46.249 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:46.251 INFO:teuthology.task.ansible.out:TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbgRgVIM6yATRN9JslmUahjLfNpF45tD1Z1'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.111115", "end": "2021-03-12 02:23:46.693639", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbgRgVIM6yATRN9JslmUahjLfNpF45tD1Z1"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:45.582524", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-12T02:23:46.252 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:46.308 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:46.309 INFO:teuthology.task.ansible.out:TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-03-12T02:23:46.312 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:46.312 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:46.340 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:46 +0000 (0:00:00.113) 0:02:26.629 ********** 2021-03-12T02:23:47.298 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-03-12T02:23:47.300 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"cache_update_time": 1615515672, "cache_updated": false, "changed": false} 2021-03-12T02:23:47.322 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:47 +0000 (0:00:00.981) 0:02:27.610 ********** 2021-03-12T02:23:47.415 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-12T02:23:47.416 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:47.439 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:47 +0000 (0:00:00.116) 0:02:27.727 ********** 2021-03-12T02:23:47.509 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-12T02:23:47.511 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbg6eVdYa1DFuagoomOAkYbgD2T0Ock6f1p'], 'uuids': ['dc6e45e0-f324-4ad6-beae-a28a75b8d58c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.128258", "end": "2021-03-12 02:23:47.968829", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbg6eVdYa1DFuagoomOAkYbgD2T0Ock6f1p"], "labels": [], "masters": [], "uuids": ["dc6e45e0-f324-4ad6-beae-a28a75b8d58c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:46.840571", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-12T02:23:47.557 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:47 +0000 (0:00:00.118) 0:02:27.846 ********** 2021-03-12T02:23:47.566 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-12T02:23:47.568 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item={'block_used': 13643750, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451358, 'size_available': 927544762368, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451358, "block_size": 4096, "block_total": 240095108, "block_used": 13643750, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544762368, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}} 2021-03-12T02:23:47.601 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:47 +0000 (0:00:00.043) 0:02:27.889 ********** 2021-03-12T02:23:47.670 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-12T02:23:47.671 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:47.797 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-12T02:23:47.799 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.011164", "end": "2021-03-12 02:23:47.645709", "rc": 0, "start": "2021-03-12 02:23:47.634545", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:47.822 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:47 +0000 (0:00:00.221) 0:02:28.111 ********** 2021-03-12T02:23:47.913 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-03-12T02:23:47.915 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'block_used': 13643750, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451358, 'size_available': 927544762368, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451358, "block_size": 4096, "block_total": 240095108, "block_used": 13643750, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544762368, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:47.951 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:47 +0000 (0:00:00.128) 0:02:28.239 ********** 2021-03-12T02:23:47.973 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:47.975 INFO:teuthology.task.ansible.out:TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbgtoJNz3OKiDDfzY7Rf8aKDT4yOd6PhK6z'], 'uuids': ['f6537e64-26b5-4e25-b34f-1f57adffedbc']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.164527", "end": "2021-03-12 02:23:48.423145", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbgtoJNz3OKiDDfzY7Rf8aKDT4yOd6PhK6z"], "labels": [], "masters": [], "uuids": ["f6537e64-26b5-4e25-b34f-1f57adffedbc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:48.258618", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.159885 s, 721 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.159885 s, 721 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:47.977 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:47.989 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:48.021 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-03-12T02:23:48.045 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:48 +0000 (0:00:00.093) 0:02:28.333 ********** 2021-03-12T02:23:48.144 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-12T02:23:48.146 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:48.272 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-12T02:23:48.273 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbgTqy1uH7IhmLis24f9lXYUn9LVSBMyFfX'], 'uuids': ['7e882b4c-578f-4f2d-9998-a2c0379877ef']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.158923", "end": "2021-03-12 02:23:48.733729", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbgTqy1uH7IhmLis24f9lXYUn9LVSBMyFfX"], "labels": [], "masters": [], "uuids": ["7e882b4c-578f-4f2d-9998-a2c0379877ef"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:48.574806", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154113 s, 748 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154113 s, 748 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:48.284 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJQN', 'wwn-0x5000c500918d87b8'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJQN-part1', 'wwn-0x5000c500918d87b8-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918d87b8', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LJQN', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJQN", "wwn-0x5000c500918d87b8"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJQN-part1", "wwn-0x5000c500918d87b8-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LJQN", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918d87b8"}}, "skip_reason": 2021-03-12T02:23:48.286 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-03-12T02:23:48.465 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-YqYRpJ-x9wa-t3Y2-i9Lc-Buje-hdmp-RsNyuB', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400ZJ400BGN', 'nvme-nvme.8086-504846543632303430305a4a34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.052980", "end": "2021-03-12 02:23:48.922350", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-YqYRpJ-x9wa-t3Y2-i9Lc-Buje-hdmp-RsNyuB", "nvme-INTEL_SSDPEDMD400G4_PHFT620400ZJ400BGN", "nvme-nvme.8086-504846543632303430305a4a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:48.869370", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0482554 s, 2.4 GB/s", "std 2021-03-12T02:23:48.467 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0482554 s, 2.4 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:48.477 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:48.479 INFO:teuthology.task.ansible.out: skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:48.767 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbg3OfBdAAd2kO1bfc9m1KicR04T5kSWM1D'], 'uuids': ['b34620f1-7a5b-4eb3-ab3c-c1c85fb239f8']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.156908", "end": "2021-03-12 02:23:49.226313", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbg3OfBdAAd2kO1bfc9m1KicR04T5kSWM1D"], "labels": [], "masters": [], "uuids": ["b34620f1-7a5b-4eb3-ab3c-c1c85fb239f8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:49.069405", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153988 s, 749 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153988 s, 749 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:48.778 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:48.790 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:49.074 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbgRgVIM6yATRN9JslmUahjLfNpF45tD1Z1'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.159029", "end": "2021-03-12 02:23:49.529416", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbgRgVIM6yATRN9JslmUahjLfNpF45tD1Z1"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:49.370387", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154201 s, 748 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154201 s, 748 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:49.086 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:49.088 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:49.371 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbg6eVdYa1DFuagoomOAkYbgD2T0Ock6f1p'], 'uuids': ['dc6e45e0-f324-4ad6-beae-a28a75b8d58c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.154786", "end": "2021-03-12 02:23:49.824557", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-MejdUpkeVhAiTSlCKKf7dSVmGzrWBTbg6eVdYa1DFuagoomOAkYbgD2T0Ock6f1p"], "labels": [], "masters": [], "uuids": ["dc6e45e0-f324-4ad6-beae-a28a75b8d58c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:49.669771", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151595 s, 761 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151595 s, 761 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:49.406 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:49 +0000 (0:00:01.361) 0:02:29.694 ********** 2021-03-12T02:23:49.774 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-03-12T02:23:49.777 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.161912", "end": "2021-03-12 02:23:50.172583", "rc": 0, "start": "2021-03-12 02:23:50.010671", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:49.797 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:49 +0000 (0:00:00.391) 0:02:30.086 ********** 2021-03-12T02:23:49.929 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-12T02:23:49.931 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIJHKnMPS5NbIMFIaM2exc7moyUcFdcSbS'], 'uuids': ['b6c79da8-a096-475e-945f-bb32de101d08']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.650700", "end": "2021-03-12 02:23:49.766395", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIJHKnMPS5NbIMFIaM2exc7moyUcFdcSbS"], "labels": [], "masters": [], "uuids": ["b6c79da8-a096-475e-945f-bb32de101d08"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:48.115695", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-12T02:23:49.931 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => 2021-03-12T02:23:49.933 INFO:teuthology.task.ansible.out:(item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:49.942 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:50.005 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.048610", "end": "2021-03-12 02:23:50.472541", "rc": 0, "start": "2021-03-12 02:23:50.423931", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:50.028 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:50 +0000 (0:00:00.230) 0:02:30.316 ********** 2021-03-12T02:23:50.097 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-03-12T02:23:50.158 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:50 +0000 (0:00:00.130) 0:02:30.447 ********** 2021-03-12T02:23:50.185 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-03-12T02:23:50.187 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:50.211 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:50 +0000 (0:00:00.052) 0:02:30.499 ********** 2021-03-12T02:23:50.248 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-03-12T02:23:50.250 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:50.272 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:50 +0000 (0:00:00.060) 0:02:30.560 ********** 2021-03-12T02:23:50.300 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-12T02:23:50.302 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:50.327 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:50 +0000 (0:00:00.055) 0:02:30.615 ********** 2021-03-12T02:23:50.364 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-12T02:23:50.366 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:50.388 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:50 +0000 (0:00:00.061) 0:02:30.677 ********** 2021-03-12T02:23:50.528 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-03-12T02:23:50.530 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-12T02:23:50.557 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:50 +0000 (0:00:00.169) 0:02:30.846 ********** 2021-03-12T02:23:50.607 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-12T02:23:50.608 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'block_used': 13643750, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451358, 'size_available': 927544762368, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451358, "block_size": 4096, "block_total": 240095108, "block_used": 13643750, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544762368, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:50.635 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:50 +0000 (0:00:00.077) 0:02:30.923 ********** 2021-03-12T02:23:50.662 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-03-12T02:23:50.664 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:50.688 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:50 +0000 (0:00:00.052) 0:02:30.976 ********** 2021-03-12T02:23:50.714 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-12T02:23:50.716 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:50.740 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:50 +0000 (0:00:00.051) 0:02:31.028 ********** 2021-03-12T02:23:50.770 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-12T02:23:50.771 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:50.792 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:50 +0000 (0:00:00.052) 0:02:31.081 ********** 2021-03-12T02:23:50.819 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-03-12T02:23:50.821 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:50.842 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:50 +0000 (0:00:00.050) 0:02:31.131 ********** 2021-03-12T02:23:50.870 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-12T02:23:50.872 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:50.895 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:50 +0000 (0:00:00.052) 0:02:31.183 ********** 2021-03-12T02:23:50.934 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-03-12T02:23:50.935 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:50.961 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:50 +0000 (0:00:00.065) 0:02:31.249 ********** 2021-03-12T02:23:51.203 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-12T02:23:51.205 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIg1Xo14I2Ikfudk0Tdeu24tgiLOwV0ZjS'], 'uuids': ['9f70978f-573c-47d0-8ba3-c6ef08dc2ac0']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.105955", "end": "2021-03-12 02:23:51.043218", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIg1Xo14I2Ikfudk0Tdeu24tgiLOwV0ZjS"], "labels": [], "masters": [], "uuids": ["9f70978f-573c-47d0-8ba3-c6ef08dc2ac0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:49.937263", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-12T02:23:51.217 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QYVM', 'wwn-0x5000c50091e2f173'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QYVM-part1', 'wwn-0x5000c50091e2f173-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50091e2f173', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5QYVM', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QYVM", "wwn-0x5000c50091e2f173"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QYVM-part1", "wwn-0x5000c50091e2f173-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QYVM", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e2f173"}}, "skip_reason": 2021-03-12T02:23:51.219 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-03-12T02:23:51.610 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-03-12T02:23:51.612 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-12T02:23:51.636 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:51 +0000 (0:00:00.675) 0:02:31.924 ********** 2021-03-12T02:23:52.299 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-12T02:23:52.301 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-12T02:23:52.544 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-12T02:23:52.546 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-qiSKgs-7QKC-yJC9-f5dO-sI9Q-LR1z-mzwoOY', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400WB400BGN', 'nvme-nvme.8086-50484654363230343030574234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.196951", "end": "2021-03-12 02:23:52.376934", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-qiSKgs-7QKC-yJC9-f5dO-sI9Q-LR1z-mzwoOY", "nvme-INTEL_SSDPEDMD400G4_PHFT620400WB400BGN", "nvme-nvme.8086-50484654363230343030574234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:51.179983", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-03-12T02:23:52.548 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:52.557 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-12T02:23:52.559 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:52.805 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-12T02:23:52.807 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-12T02:23:53.046 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-12T02:23:53.310 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-12T02:23:53.352 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:53 +0000 (0:00:01.716) 0:02:33.640 ********** 2021-03-12T02:23:53.522 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-12T02:23:53.524 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-12T02:23:53.545 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:53 +0000 (0:00:00.193) 0:02:33.833 ********** 2021-03-12T02:23:53.711 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-12T02:23:53.713 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-12T02:23:53.817 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-12T02:23:53.819 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIJqPJBVXT49ZZOtIfku7vDmOv4vFxf2Th'], 'uuids': ['9776cd0c-2c08-4e0e-b546-acc8ba8dc152']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.125167", "end": "2021-03-12 02:23:53.650781", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIJqPJBVXT49ZZOtIfku7vDmOv4vFxf2Th"], "labels": [], "masters": [], "uuids": ["9776cd0c-2c08-4e0e-b546-acc8ba8dc152"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:52.525614", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi114.front.sepia.ceph.com] => (i 2021-03-12T02:23:53.821 INFO:teuthology.task.ansible.out:tem={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:53.831 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-12T02:23:53.832 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:53.948 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-12T02:23:53.949 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-12T02:23:54.064 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-12T02:23:54.076 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:54.102 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:54 +0000 (0:00:00.557) 0:02:34.391 ********** 2021-03-12T02:23:54.144 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-03-12T02:23:54.146 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:54.171 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:54 +0000 (0:00:00.067) 0:02:34.459 ********** 2021-03-12T02:23:54.202 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:54.225 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:54 +0000 (0:00:00.054) 0:02:34.513 ********** 2021-03-12T02:23:54.373 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-03-12T02:23:54.374 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-03-12T02:23:54.398 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:54 +0000 (0:00:00.173) 0:02:34.687 ********** 2021-03-12T02:23:54.437 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-03-12T02:23:54.438 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-12T02:23:54.471 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:54 +0000 (0:00:00.068) 0:02:34.755 ********** 2021-03-12T02:23:55.050 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi077.front.sepia.ceph.com] => {"changed": true} 2021-03-12T02:23:55.076 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:55 +0000 (0:00:00.608) 0:02:35.364 ********** 2021-03-12T02:23:55.084 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-12T02:23:55.085 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIaWDN9jypzsVmTwycb1S570W5dHH6lEjJ'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.120192", "end": "2021-03-12 02:23:54.918606", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIaWDN9jypzsVmTwycb1S570W5dHH6lEjJ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:53.798414", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-12T02:23:55.086 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:55.570 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-03-12T02:23:55.572 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-03-12T02:23:55.593 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:55 +0000 (0:00:00.517) 0:02:35.881 ********** 2021-03-12T02:23:55.621 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-03-12T02:23:55.622 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:55.645 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:55 +0000 (0:00:00.052) 0:02:35.934 ********** 2021-03-12T02:23:55.684 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:55.709 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:55 +0000 (0:00:00.063) 0:02:35.998 ********** 2021-03-12T02:23:55.738 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-12T02:23:55.740 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:55.764 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:55 +0000 (0:00:00.054) 0:02:36.052 ********** 2021-03-12T02:23:55.792 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-12T02:23:55.794 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:55.816 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:55 +0000 (0:00:00.051) 0:02:36.104 ********** 2021-03-12T02:23:55.980 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-03-12T02:23:56.006 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:56 +0000 (0:00:00.189) 0:02:36.294 ********** 2021-03-12T02:23:56.162 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-03-12T02:23:56.163 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-03-12T02:23:56.189 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:56 +0000 (0:00:00.183) 0:02:36.477 ********** 2021-03-12T02:23:56.491 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-12T02:23:56.494 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIbPLfPV3pVgcvy9y3ftYcM1I1QtnQ6WYn'], 'uuids': ['5ff140d1-160f-416b-ac07-37514e64981e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.117200", "end": "2021-03-12 02:23:56.167857", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIbPLfPV3pVgcvy9y3ftYcM1I1QtnQ6WYn"], "labels": [], "masters": [], "uuids": ["5ff140d1-160f-416b-ac07-37514e64981e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:55.050657", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-12T02:23:56.541 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:56 +0000 (0:00:00.351) 0:02:36.829 ********** 2021-03-12T02:23:56.607 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-12T02:23:56.609 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:56.715 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-03-12T02:23:56.716 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-03-12T02:23:56.740 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:56 +0000 (0:00:00.198) 0:02:37.028 ********** 2021-03-12T02:23:56.895 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:56.896 INFO:teuthology.task.ansible.out:TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-12T02:23:56.898 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:56.899 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIJHKnMPS5NbIMFIaM2exc7moyUcFdcSbS'], 'uuids': ['b6c79da8-a096-475e-945f-bb32de101d08']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.156823", "end": "2021-03-12 02:23:56.735294", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIJHKnMPS5NbIMFIaM2exc7moyUcFdcSbS"], "labels": [], "masters": [], "uuids": ["b6c79da8-a096-475e-945f-bb32de101d08"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:56.578471", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.152064 s, 759 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.152064 s, 759 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:56.901 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:56.912 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:57.100 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-03-12T02:23:57.102 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-03-12T02:23:57.125 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:57 +0000 (0:00:00.385) 0:02:37.413 ********** 2021-03-12T02:23:57.205 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-12T02:23:57.206 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIg1Xo14I2Ikfudk0Tdeu24tgiLOwV0ZjS'], 'uuids': ['9f70978f-573c-47d0-8ba3-c6ef08dc2ac0']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.158483", "end": "2021-03-12 02:23:57.041722", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIg1Xo14I2Ikfudk0Tdeu24tgiLOwV0ZjS"], "labels": [], "masters": [], "uuids": ["9f70978f-573c-47d0-8ba3-c6ef08dc2ac0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:56.883239", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153674 s, 751 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153674 s, 751 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:57.218 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QYVM', 'wwn-0x5000c50091e2f173'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QYVM-part1', 'wwn-0x5000c50091e2f173-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50091e2f173', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5QYVM', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QYVM", "wwn-0x5000c50091e2f173"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QYVM-part1", "wwn-0x5000c50091e2f173-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QYVM", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e2f173"}}, "skip_reason": 2021-03-12T02:23:57.220 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-03-12T02:23:57.398 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-qiSKgs-7QKC-yJC9-f5dO-sI9Q-LR1z-mzwoOY', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400WB400BGN', 'nvme-nvme.8086-50484654363230343030574234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.052563", "end": "2021-03-12 02:23:57.239919", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-qiSKgs-7QKC-yJC9-f5dO-sI9Q-LR1z-mzwoOY", "nvme-INTEL_SSDPEDMD400G4_PHFT620400WB400BGN", "nvme-nvme.8086-50484654363230343030574234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:57.187356", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0478097 s, 2.4 GB/s", "std 2021-03-12T02:23:57.400 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0478097 s, 2.4 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:57.410 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:57.412 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:57.422 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:57.718 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIJqPJBVXT49ZZOtIfku7vDmOv4vFxf2Th'], 'uuids': ['9776cd0c-2c08-4e0e-b546-acc8ba8dc152']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.153161", "end": "2021-03-12 02:23:57.556074", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIJqPJBVXT49ZZOtIfku7vDmOv4vFxf2Th"], "labels": [], "masters": [], "uuids": ["9776cd0c-2c08-4e0e-b546-acc8ba8dc152"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:57.402913", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148539 s, 777 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148539 s, 777 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:57.730 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:57.742 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:57.996 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-03-12T02:23:57.998 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"cache_update_time": 1615515667, "cache_updated": false, "changed": false} 2021-03-12T02:23:58.025 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:58 +0000 (0:00:00.899) 0:02:38.313 ********** 2021-03-12T02:23:58.044 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-12T02:23:58.046 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIaWDN9jypzsVmTwycb1S570W5dHH6lEjJ'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.160479", "end": "2021-03-12 02:23:57.882086", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIaWDN9jypzsVmTwycb1S570W5dHH6lEjJ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:57.721607", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.155598 s, 741 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.155598 s, 741 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:58.060 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:58.221 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-03-12T02:23:58.223 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.046231", "end": "2021-03-12 02:23:58.682878", "rc": 0, "start": "2021-03-12 02:23:58.636647", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-03-12T02:23:58.246 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:58 +0000 (0:00:00.220) 0:02:38.534 ********** 2021-03-12T02:23:58.286 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-03-12T02:23:58.288 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:58.313 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:58 +0000 (0:00:00.066) 0:02:38.601 ********** 2021-03-12T02:23:58.353 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-12T02:23:58.354 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIbPLfPV3pVgcvy9y3ftYcM1I1QtnQ6WYn'], 'uuids': ['5ff140d1-160f-416b-ac07-37514e64981e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.155842", "end": "2021-03-12 02:23:58.182911", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-Q3VtWyRgaYNf6uXC4kyFVmfuRbZVNTrIbPLfPV3pVgcvy9y3ftYcM1I1QtnQ6WYn"], "labels": [], "masters": [], "uuids": ["5ff140d1-160f-416b-ac07-37514e64981e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-12 02:23:58.027069", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150942 s, 764 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150942 s, 764 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:58.356 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-03-12T02:23:58.358 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:58.366 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-12T02:23:58.408 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:58 +0000 (0:00:00.095) 0:02:38.696 ********** 2021-03-12T02:23:58.432 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:58 +0000 (0:00:00.023) 0:02:38.720 ********** 2021-03-12T02:23:58.722 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi114.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.137952", "end": "2021-03-12 02:23:58.556882", "rc": 0, "start": "2021-03-12 02:23:58.418930", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:58.749 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:58 +0000 (0:00:00.317) 0:02:39.038 ********** 2021-03-12T02:23:58.949 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.049022", "end": "2021-03-12 02:23:58.795025", "rc": 0, "start": "2021-03-12 02:23:58.746003", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:23:58.969 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:58 +0000 (0:00:00.219) 0:02:39.257 ********** 2021-03-12T02:23:59.006 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-03-12T02:23:59.036 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:59 +0000 (0:00:00.066) 0:02:39.324 ********** 2021-03-12T02:23:59.078 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-03-12T02:23:59.080 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:59.109 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:59 +0000 (0:00:00.073) 0:02:39.397 ********** 2021-03-12T02:23:59.158 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:59.160 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:59.181 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:59 +0000 (0:00:00.072) 0:02:39.470 ********** 2021-03-12T02:23:59.219 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-12T02:23:59.221 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:59.247 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:59 +0000 (0:00:00.065) 0:02:39.535 ********** 2021-03-12T02:23:59.285 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-12T02:23:59.289 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:59.315 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:59 +0000 (0:00:00.067) 0:02:39.603 ********** 2021-03-12T02:23:59.335 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-03-12T02:23:59.336 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"cache_update_time": 1615515667, "cache_updated": false, "changed": false} 2021-03-12T02:23:59.381 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:59 +0000 (0:00:00.065) 0:02:39.669 ********** 2021-03-12T02:23:59.426 INFO:teuthology.task.ansible.out: 2021-03-12T02:23:59.428 INFO:teuthology.task.ansible.out:TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-03-12T02:23:59.432 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:59.459 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:59 +0000 (0:00:00.078) 0:02:39.747 ********** 2021-03-12T02:23:59.474 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-03-12T02:23:59.475 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-12T02:23:59.509 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:59 +0000 (0:00:00.049) 0:02:39.797 ********** 2021-03-12T02:23:59.589 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-12T02:23:59.591 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'block_used': 13643750, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451358, 'size_available': 927544762368, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451358, "block_size": 4096, "block_total": 240095108, "block_used": 13643750, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927544762368, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:59.638 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:59 +0000 (0:00:00.129) 0:02:39.926 ********** 2021-03-12T02:23:59.676 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-03-12T02:23:59.678 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:59.704 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:59 +0000 (0:00:00.065) 0:02:39.992 ********** 2021-03-12T02:23:59.731 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-12T02:23:59.733 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:59.755 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:59 +0000 (0:00:00.050) 0:02:40.043 ********** 2021-03-12T02:23:59.784 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-12T02:23:59.786 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:59.809 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:59 +0000 (0:00:00.054) 0:02:40.097 ********** 2021-03-12T02:23:59.827 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-03-12T02:23:59.829 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-03-12T02:23:59.858 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:59 +0000 (0:00:00.049) 0:02:40.147 ********** 2021-03-12T02:23:59.870 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-03-12T02:23:59.872 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:59.908 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:59 +0000 (0:00:00.049) 0:02:40.197 ********** 2021-03-12T02:23:59.917 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-03-12T02:23:59.919 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:59.944 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:59 +0000 (0:00:00.034) 0:02:40.231 ********** 2021-03-12T02:23:59.950 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-12T02:23:59.952 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:23:59.980 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:23:59 +0000 (0:00:00.035) 0:02:40.267 ********** 2021-03-12T02:24:00.008 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:00.039 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:00 +0000 (0:00:00.059) 0:02:40.327 ********** 2021-03-12T02:24:00.183 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-03-12T02:24:00.184 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-03-12 02:20:23 UTC", "ActiveEnterTimestampMonotonic": "16986493", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target systemd-tmpfiles-setup.service systemd-journald.socket system.slice -.mount network.target basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-03-12 02:20:23 UTC", "AssertTimestampMonotonic": "16880139", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-03-12 02:20:23 UTC", "ConditionTimestampMonotonic": "16880139", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-03-12T02:24:00.185 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1120", "ExecMainStartTimestamp": "Fri 2021-03-12 02:20:23 UTC", "ExecMainStartTimestampMonotonic": "16986475", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-03-12 02:20:23 UTC", "InactiveExitTimestampMonotonic": "16880917", "InvocationID": "3b9a5aa7542f40e6a343b061771ee595", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-03-12T02:24:00.187 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1120", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-03-12 02:20:23 UTC", "S 2021-03-12T02:24:00.189 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "16986493", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-03-12 02:20:23 UTC", "WatchdogTimestampMonotonic": "16986491", "WatchdogUSec": "0"}} 2021-03-12T02:24:00.212 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:00 +0000 (0:00:00.172) 0:02:40.499 ********** 2021-03-12T02:24:00.240 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-12T02:24:00.267 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:00 +0000 (0:00:00.056) 0:02:40.556 ********** 2021-03-12T02:24:00.297 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-12T02:24:00.299 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:00.324 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:00 +0000 (0:00:00.056) 0:02:40.613 ********** 2021-03-12T02:24:00.354 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-03-12T02:24:00.356 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:00.379 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:00 +0000 (0:00:00.054) 0:02:40.667 ********** 2021-03-12T02:24:00.385 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-03-12T02:24:00.387 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-12T02:24:00.413 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:00 +0000 (0:00:00.034) 0:02:40.701 ********** 2021-03-12T02:24:00.731 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-12T02:24:00.733 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-12T02:24:00.983 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-12T02:24:01.218 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-03-12T02:24:01.220 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"cache_update_time": 1615515667, "cache_updated": false, "changed": false} 2021-03-12T02:24:01.245 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:01 +0000 (0:00:00.832) 0:02:41.534 ********** 2021-03-12T02:24:01.257 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-12T02:24:01.259 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-12T02:24:01.280 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-03-12T02:24:01.282 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:01.309 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:01 +0000 (0:00:00.063) 0:02:41.598 ********** 2021-03-12T02:24:01.445 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-03-12T02:24:01.470 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:01 +0000 (0:00:00.161) 0:02:41.759 ********** 2021-03-12T02:24:01.500 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-12T02:24:01.501 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-12T02:24:01.746 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-12T02:24:01.772 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:01 +0000 (0:00:00.301) 0:02:42.060 ********** 2021-03-12T02:24:01.828 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-03-12T02:24:01.830 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-03-12T02:24:01.855 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:01 +0000 (0:00:00.083) 0:02:42.144 ********** 2021-03-12T02:24:01.917 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-12T02:24:01.919 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-12T02:24:01.942 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:01 +0000 (0:00:00.086) 0:02:42.230 ********** 2021-03-12T02:24:02.117 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-12T02:24:02.118 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-12T02:24:02.235 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-12T02:24:02.361 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-12T02:24:02.479 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-12T02:24:02.491 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:02.493 INFO:teuthology.task.ansible.out: 2021-03-12T02:24:02.521 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:02 +0000 (0:00:00.578) 0:02:42.809 ********** 2021-03-12T02:24:02.561 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-03-12T02:24:02.563 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:02.585 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:02 +0000 (0:00:00.063) 0:02:42.873 ********** 2021-03-12T02:24:02.612 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-03-12T02:24:02.614 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:02.636 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:02 +0000 (0:00:00.051) 0:02:42.925 ********** 2021-03-12T02:24:02.772 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-03-12T02:24:02.773 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-03-12T02:24:02.796 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:02 +0000 (0:00:00.159) 0:02:43.084 ********** 2021-03-12T02:24:02.802 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-03-12T02:24:02.804 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-03-12T02:24:02.829 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:02 +0000 (0:00:00.033) 0:02:43.117 ********** 2021-03-12T02:24:02.866 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi114.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-12T02:24:02.891 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:02 +0000 (0:00:00.061) 0:02:43.179 ********** 2021-03-12T02:24:03.136 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-03-12T02:24:03.140 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true} 2021-03-12T02:24:03.166 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:03 +0000 (0:00:00.275) 0:02:43.454 ********** 2021-03-12T02:24:03.309 INFO:teuthology.task.ansible.out: 2021-03-12T02:24:03.311 INFO:teuthology.task.ansible.out:TASK [testnode : Purge resolvconf] ********************************************* ok: [smithi077.front.sepia.ceph.com] => {"changed": false} 2021-03-12T02:24:03.340 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:03 +0000 (0:00:00.172) 0:02:43.626 ********** 2021-03-12T02:24:03.373 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-03-12T02:24:03.399 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:03 +0000 (0:00:00.061) 0:02:43.688 ********** 2021-03-12T02:24:03.556 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-03-12T02:24:03.557 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1615515644.6449816, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1615515644.6409817, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 262873, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1615515644.6409817, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3476867264", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-03-12T02:24:03.581 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:03 +0000 (0:00:00.181) 0:02:43.869 ********** 2021-03-12T02:24:03.879 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-03-12T02:24:03.882 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-03-12T02:24:03.916 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:03 +0000 (0:00:00.334) 0:02:44.204 ********** 2021-03-12T02:24:03.956 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-03-12T02:24:03.958 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-12T02:24:03.980 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:03 +0000 (0:00:00.063) 0:02:44.268 ********** 2021-03-12T02:24:04.018 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-03-12T02:24:04.020 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.041 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:04 +0000 (0:00:00.061) 0:02:44.329 ********** 2021-03-12T02:24:04.185 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-03-12T02:24:04.187 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-12T02:24:04.212 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:04 +0000 (0:00:00.170) 0:02:44.500 ********** 2021-03-12T02:24:04.355 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-03-12T02:24:04.356 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-12T02:24:04.387 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:04 +0000 (0:00:00.174) 0:02:44.675 ********** 2021-03-12T02:24:04.396 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-03-12T02:24:04.398 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-03-12T02:24:04.424 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:04 +0000 (0:00:00.035) 0:02:44.711 ********** 2021-03-12T02:24:04.430 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-03-12T02:24:04.432 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.468 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:04 +0000 (0:00:00.045) 0:02:44.756 ********** 2021-03-12T02:24:04.477 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-03-12T02:24:04.478 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.507 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:04 +0000 (0:00:00.039) 0:02:44.796 ********** 2021-03-12T02:24:04.518 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-03-12T02:24:04.520 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.523 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.537 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.539 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.541 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.572 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:04 +0000 (0:00:00.064) 0:02:44.861 ********** 2021-03-12T02:24:04.597 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:04 +0000 (0:00:00.024) 0:02:44.886 ********** 2021-03-12T02:24:04.617 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-03-12T02:24:04.619 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.630 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.632 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.633 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi077.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.659 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:04 +0000 (0:00:00.061) 0:02:44.947 ********** 2021-03-12T02:24:04.685 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:04 +0000 (0:00:00.026) 0:02:44.974 ********** 2021-03-12T02:24:04.692 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-12T02:24:04.693 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.720 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:04 +0000 (0:00:00.034) 0:02:45.008 ********** 2021-03-12T02:24:04.727 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-12T02:24:04.729 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.752 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:04 +0000 (0:00:00.032) 0:02:45.041 ********** 2021-03-12T02:24:04.763 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-03-12T02:24:04.765 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.793 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:04 +0000 (0:00:00.040) 0:02:45.081 ********** 2021-03-12T02:24:04.820 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-03-12T02:24:04.822 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.847 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:04 +0000 (0:00:00.053) 0:02:45.135 ********** 2021-03-12T02:24:04.876 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.899 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:04 +0000 (0:00:00.052) 0:02:45.187 ********** 2021-03-12T02:24:04.916 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-03-12T02:24:04.918 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-03-12T02:24:04.940 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:04 +0000 (0:00:00.041) 0:02:45.229 ********** 2021-03-12T02:24:04.947 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-03-12T02:24:04.949 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:04.984 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:04 +0000 (0:00:00.043) 0:02:45.273 ********** 2021-03-12T02:24:05.014 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-12T02:24:05.016 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:05.043 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:05 +0000 (0:00:00.058) 0:02:45.331 ********** 2021-03-12T02:24:05.070 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-03-12T02:24:05.072 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:05.103 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:05 +0000 (0:00:00.059) 0:02:45.391 ********** 2021-03-12T02:24:05.112 INFO:teuthology.task.ansible.out: 2021-03-12T02:24:05.113 INFO:teuthology.task.ansible.out:TASK [testnode : Add CPAN config directory for the root user.] ***************** ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-03-12T02:24:05.141 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:05 +0000 (0:00:00.038) 0:02:45.430 ********** 2021-03-12T02:24:05.148 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:05.174 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:05 +0000 (0:00:00.032) 0:02:45.462 ********** 2021-03-12T02:24:05.350 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-03-12T02:24:05.352 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010798", "end": "2021-03-12 02:24:05.314290", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-12 02:24:05.303492", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:24:05.467 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006005", "end": "2021-03-12 02:24:05.443649", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-12 02:24:05.437644", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:24:05.515 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-03-12T02:24:05.540 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:05 +0000 (0:00:00.365) 0:02:45.828 ********** 2021-03-12T02:24:05.600 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-03-12T02:24:05.602 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005473", "end": "2021-03-12 02:24:05.565478", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-12 02:24:05.560005", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-12T02:24:05.824 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.101772", "end": "2021-03-12 02:24:05.786411", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-12 02:24:05.684639", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:24:05.866 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:05 +0000 (0:00:00.325) 0:02:46.154 ********** 2021-03-12T02:24:05.894 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-12T02:24:05.917 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:05 +0000 (0:00:00.051) 0:02:46.205 ********** 2021-03-12T02:24:05.924 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-03-12T02:24:05.926 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-03-12T02:24:05.952 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:05 +0000 (0:00:00.034) 0:02:46.240 ********** 2021-03-12T02:24:06.053 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-03-12T02:24:06.861 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** ok: [smithi114.front.sepia.ceph.com] => {"cache_update_time": 1615515672, "cache_updated": false, "changed": false} 2021-03-12T02:24:06.884 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:06 +0000 (0:00:00.932) 0:02:47.172 ********** 2021-03-12T02:24:07.068 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-03-12T02:24:07.071 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.044229", "end": "2021-03-12 02:24:07.038643", "rc": 0, "start": "2021-03-12 02:24:06.994414", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-03-12T02:24:07.092 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:07 +0000 (0:00:00.207) 0:02:47.380 ********** 2021-03-12T02:24:07.130 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:07.155 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:07 +0000 (0:00:00.062) 0:02:47.442 ********** 2021-03-12T02:24:07.183 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:07.206 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:07 +0000 (0:00:00.051) 0:02:47.494 ********** 2021-03-12T02:24:08.086 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-03-12T02:24:08.087 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"cache_update_time": 1615515672, "cache_updated": false, "changed": false} 2021-03-12T02:24:08.118 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:08 +0000 (0:00:00.912) 0:02:48.406 ********** 2021-03-12T02:24:08.146 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-03-12T02:24:08.148 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:08.176 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:08 +0000 (0:00:00.058) 0:02:48.465 ********** 2021-03-12T02:24:08.552 INFO:teuthology.task.ansible.out: 2021-03-12T02:24:08.553 INFO:teuthology.task.ansible.out:TASK [testnode : Create the ntp.conf file.] ************************************ 2021-03-12T02:24:08.553 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-03-12T02:24:08.557 INFO:teuthology.task.ansible.out: 2021-03-12T02:24:08.599 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:08 +0000 (0:00:00.422) 0:02:48.887 ********** 2021-03-12T02:24:08.601 INFO:teuthology.task.ansible.out: 2021-03-12T02:24:08.632 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-03-12T02:24:08.633 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:08.666 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:08 +0000 (0:00:00.067) 0:02:48.954 ********** 2021-03-12T02:24:08.891 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-03-12T02:24:08.893 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-03-12 02:20:20 UTC", "ActiveEnterTimestampMonotonic": "16330948", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target network.target system.slice systemd-journald.socket basic.target systemd-tmpfiles-setup.service -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-03-12 02:20:19 UTC", "AssertTimestampMonotonic": "15744121", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-03-12 02:20:19 UTC", "ConditionTimestampMonotonic": "15744121", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-03-12T02:24:08.894 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1122", "ExecMainStartTimestamp": "Fri 2021-03-12 02:20:20 UTC", "ExecMainStartTimestampMonotonic": "16330926", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-03-12 02:20:19 UTC", "InactiveExitTimestampMonotonic": "15744974", "InvocationID": "b7c1d5945c7143d39162111b133e51f2", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-03-12T02:24:08.896 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1122", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-03-12 02:20:20 UTC", "S 2021-03-12T02:24:08.897 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "16330948", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-03-12 02:20:20 UTC", "WatchdogTimestampMonotonic": "16330945", "WatchdogUSec": "0"}} 2021-03-12T02:24:08.920 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:08 +0000 (0:00:00.254) 0:02:49.208 ********** 2021-03-12T02:24:08.922 INFO:teuthology.task.ansible.out: 2021-03-12T02:24:08.958 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-12T02:24:08.960 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-12T02:24:08.985 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:08 +0000 (0:00:00.064) 0:02:49.273 ********** 2021-03-12T02:24:09.019 INFO:teuthology.task.ansible.out: 2021-03-12T02:24:09.021 INFO:teuthology.task.ansible.out:TASK [testnode : set_fact] ***************************************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:09.052 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:09 +0000 (0:00:00.066) 0:02:49.340 ********** 2021-03-12T02:24:09.081 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-03-12T02:24:09.083 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:09.107 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:09 +0000 (0:00:00.054) 0:02:49.394 ********** 2021-03-12T02:24:10.009 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-03-12T02:24:10.011 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"cache_update_time": 1615515672, "cache_updated": false, "changed": false} 2021-03-12T02:24:10.033 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:10 +0000 (0:00:00.927) 0:02:50.322 ********** 2021-03-12T02:24:10.062 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-03-12T02:24:10.064 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:10.088 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:10 +0000 (0:00:00.051) 0:02:50.373 ********** 2021-03-12T02:24:10.228 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-03-12T02:24:10.230 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-03-12T02:24:10.254 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:10 +0000 (0:00:00.168) 0:02:50.542 ********** 2021-03-12T02:24:10.667 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-03-12T02:24:10.693 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:10 +0000 (0:00:00.439) 0:02:50.981 ********** 2021-03-12T02:24:11.483 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-03-12T02:24:11.507 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:11 +0000 (0:00:00.813) 0:02:51.795 ********** 2021-03-12T02:24:12.086 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-03-12T02:24:12.088 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false} 2021-03-12T02:24:12.115 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:12 +0000 (0:00:00.608) 0:02:52.403 ********** 2021-03-12T02:24:12.154 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-03-12T02:24:12.156 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-03-12T02:24:12.177 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:12 +0000 (0:00:00.062) 0:02:52.466 ********** 2021-03-12T02:24:12.332 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-03-12T02:24:12.333 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1615515641.120028, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1615515641.116028, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 262873, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1615515641.116028, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3476867264", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-03-12T02:24:12.356 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:12 +0000 (0:00:00.178) 0:02:52.644 ********** 2021-03-12T02:24:12.521 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-03-12T02:24:12.523 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-03-12T02:24:12.546 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:12 +0000 (0:00:00.189) 0:02:52.834 ********** 2021-03-12T02:24:12.587 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi114.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-12T02:24:12.610 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:12 +0000 (0:00:00.062) 0:02:52.896 ********** 2021-03-12T02:24:12.640 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-03-12T02:24:12.643 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:12.665 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:12 +0000 (0:00:00.057) 0:02:52.954 ********** 2021-03-12T02:24:12.820 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-03-12T02:24:12.820 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-12T02:24:12.847 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:12 +0000 (0:00:00.181) 0:02:53.135 ********** 2021-03-12T02:24:12.987 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-03-12T02:24:12.989 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-12T02:24:13.031 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:13 +0000 (0:00:00.183) 0:02:53.319 ********** 2021-03-12T02:24:13.101 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-03-12T02:24:13.102 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:13.130 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:13 +0000 (0:00:00.099) 0:02:53.418 ********** 2021-03-12T02:24:13.211 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-03-12T02:24:13.213 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-03-12T02:24:13.223 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-03-12T02:24:13.224 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-03-12T02:24:13.234 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-12T02:24:13.245 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-03-12T02:24:13.292 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:13 +0000 (0:00:00.158) 0:02:53.576 ********** 2021-03-12T02:24:13.359 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-03-12T02:24:13.361 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi114.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:13.371 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:13.393 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:13.395 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:13.418 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:13 +0000 (0:00:00.130) 0:02:53.706 ********** 2021-03-12T02:24:13.459 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-12T02:24:13.461 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:13.496 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:13 +0000 (0:00:00.078) 0:02:53.785 ********** 2021-03-12T02:24:13.535 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-03-12T02:24:13.537 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:13.570 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:13 +0000 (0:00:00.073) 0:02:53.859 ********** 2021-03-12T02:24:13.618 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-03-12T02:24:13.620 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:13.647 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:13 +0000 (0:00:00.076) 0:02:53.935 ********** 2021-03-12T02:24:13.677 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-03-12T02:24:13.679 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:13.701 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:13 +0000 (0:00:00.054) 0:02:53.989 ********** 2021-03-12T02:24:13.705 INFO:teuthology.task.ansible.out: 2021-03-12T02:24:13.765 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:13.793 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:13 +0000 (0:00:00.091) 0:02:54.081 ********** 2021-03-12T02:24:13.833 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-12T02:24:13.834 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:13.878 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:13 +0000 (0:00:00.084) 0:02:54.165 ********** 2021-03-12T02:24:13.918 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-03-12T02:24:13.920 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:13.958 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:13 +0000 (0:00:00.081) 0:02:54.247 ********** 2021-03-12T02:24:14.000 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-12T02:24:14.001 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:14.026 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:14 +0000 (0:00:00.067) 0:02:54.315 ********** 2021-03-12T02:24:14.219 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-03-12T02:24:14.221 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.009928", "end": "2021-03-12 02:24:14.163779", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-12 02:24:14.153851", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:24:14.337 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006076", "end": "2021-03-12 02:24:14.305890", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-12 02:24:14.299814", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:24:14.463 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005502", "end": "2021-03-12 02:24:14.436687", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-12 02:24:14.431185", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-12T02:24:14.600 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.019214", "end": "2021-03-12 02:24:14.570575", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-12 02:24:14.551361", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:24:14.639 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:14 +0000 (0:00:00.612) 0:02:54.927 ********** 2021-03-12T02:24:14.693 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-12T02:24:14.719 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:14 +0000 (0:00:00.080) 0:02:55.007 ********** 2021-03-12T02:24:14.887 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-03-12T02:24:14.889 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-03-12T02:24:14.970 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-03-12T02:24:14.996 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:14 +0000 (0:00:00.277) 0:02:55.285 ********** 2021-03-12T02:24:15.022 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:15 +0000 (0:00:00.025) 0:02:55.310 ********** 2021-03-12T02:24:15.778 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi077.front.sepia.ceph.com] 2021-03-12T02:24:15.813 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] 2021-03-12T02:24:15.854 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:15 +0000 (0:00:00.831) 0:02:56.142 ********** 2021-03-12T02:24:15.902 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:15 +0000 (0:00:00.048) 0:02:56.190 ********** 2021-03-12T02:24:15.909 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-12T02:24:15.910 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-03-12T02:24:15.935 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:15 +0000 (0:00:00.032) 0:02:56.223 ********** 2021-03-12T02:24:15.942 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-03-12T02:24:15.968 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:15 +0000 (0:00:00.032) 0:02:56.256 ********** 2021-03-12T02:24:15.977 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-12T02:24:15.978 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:16.006 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:16 +0000 (0:00:00.037) 0:02:56.294 ********** 2021-03-12T02:24:16.014 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:16.041 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:16 +0000 (0:00:00.035) 0:02:56.329 ********** 2021-03-12T02:24:16.071 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** ok: [smithi077.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-03-12T02:24:16.096 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:16 +0000 (0:00:00.055) 0:02:56.384 ********** 2021-03-12T02:24:16.104 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-03-12T02:24:16.131 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:16 +0000 (0:00:00.035) 0:02:56.419 ********** 2021-03-12T02:24:16.133 INFO:teuthology.task.ansible.out: 2021-03-12T02:24:17.049 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-03-12T02:24:17.051 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"cache_update_time": 1615515672, "cache_updated": false, "changed": false} ok: [smithi077.front.sepia.ceph.com] => {"cache_update_time": 1615515667, "cache_updated": false, "changed": false} 2021-03-12T02:24:17.075 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:17 +0000 (0:00:00.944) 0:02:57.364 ********** 2021-03-12T02:24:17.105 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:17 +0000 (0:00:00.029) 0:02:57.394 ********** 2021-03-12T02:24:17.114 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-12T02:24:17.116 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:17.144 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:17 +0000 (0:00:00.038) 0:02:57.432 ********** 2021-03-12T02:24:17.151 INFO:teuthology.task.ansible.out:skipping: [smithi114.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-12T02:24:17.181 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:17 +0000 (0:00:00.037) 0:02:57.469 ********** 2021-03-12T02:24:17.189 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-12T02:24:17.190 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-03-12T02:24:17.219 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:17 +0000 (0:00:00.038) 0:02:57.507 ********** 2021-03-12T02:24:17.227 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-03-12T02:24:17.253 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:17 +0000 (0:00:00.033) 0:02:57.541 ********** 2021-03-12T02:24:17.407 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-03-12T02:24:17.408 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-12T02:24:17.431 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-12T02:24:17.527 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-12T02:24:17.569 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:17 +0000 (0:00:00.316) 0:02:57.857 ********** 2021-03-12T02:24:17.578 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-12T02:24:17.606 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:17 +0000 (0:00:00.036) 0:02:57.894 ********** 2021-03-12T02:24:18.010 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-03-12T02:24:18.012 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615515857.6830726-32530-5501863382975/source", "state": "file", "uid": 0} 2021-03-12T02:24:18.022 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615515857.6603665-32525-185402251356364/source", "state": "file", "uid": 0} 2021-03-12T02:24:18.358 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615515858.0375519-32525-29095020240634/source", "state": "file", "uid": 0} 2021-03-12T02:24:18.360 INFO:teuthology.task.ansible.out: 2021-03-12T02:24:18.393 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:18 +0000 (0:00:00.786) 0:02:58.681 ********** 2021-03-12T02:24:18.401 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615515858.032558-32530-67100092717286/source", "state": "file", "uid": 0} 2021-03-12T02:24:18.430 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:18 +0000 (0:00:00.037) 0:02:58.718 ********** 2021-03-12T02:24:21.895 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-12T02:24:21.897 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-JoIXd4-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-JoIXd4-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-03-12T02:24:21.899 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-03-12T02:24:21.921 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:21 +0000 (0:00:03.490) 0:03:02.209 ********** 2021-03-12T02:24:22.065 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-12T02:24:22.090 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:22 +0000 (0:00:00.169) 0:03:02.378 ********** 2021-03-12T02:24:22.169 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-12T02:24:22.171 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-f5jp7n-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-f5jp7n-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-03-12T02:24:22.173 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-03-12T02:24:22.195 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:22 +0000 (0:00:00.105) 0:03:02.483 ********** 2021-03-12T02:24:22.337 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-03-12T02:24:22.338 INFO:teuthology.task.ansible.out:ok: [smithi114.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-12T02:24:22.361 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:22 +0000 (0:00:00.165) 0:03:02.649 ********** 2021-03-12T02:24:22.442 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615515862.1499984-584-119171676475254/source", "state": "file", "uid": 0} 2021-03-12T02:24:22.468 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:22 +0000 (0:00:00.107) 0:03:02.756 ********** 2021-03-12T02:24:22.730 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615515862.4185777-609-734646123519/source", "state": "file", "uid": 0} 2021-03-12T02:24:22.731 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.112407", "end": "2021-03-12 02:24:22.708112", "rc": 0, "start": "2021-03-12 02:24:22.595705", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:24:22.754 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:22 +0000 (0:00:00.286) 0:03:03.043 ********** 2021-03-12T02:24:22.780 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:22 +0000 (0:00:00.025) 0:03:03.068 ********** 2021-03-12T02:24:23.141 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.197435", "end": "2021-03-12 02:24:23.106560", "rc": 0, "start": "2021-03-12 02:24:22.909125", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-12T02:24:23.166 INFO:teuthology.task.ansible.out:Friday 12 March 2021 02:24:23 +0000 (0:00:00.385) 0:03:03.454 ********** 2021-03-12T02:24:25.789 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-03-12T02:24:25.790 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "systemd-journald.socket firewalld.service sysinit.target system.slice basic.target network-online.target docker.socket containerd.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-03-12T02:24:25.792 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "12802 2021-03-12T02:24:25.794 INFO:teuthology.task.ansible.out:9", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target docker.socket", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-03-12 02:20:47 UTC", "StateChangeTimestampMonotonic": "40899130", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "Su 2021-03-12T02:24:25.796 INFO:teuthology.task.ansible.out:ccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-12T02:24:26.421 INFO:teuthology.task.ansible.out:changed: [smithi114.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "network-online.target basic.target firewalld.service sysinit.target systemd-journald.socket docker.socket system.slice containerd.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-03-12T02:24:26.423 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "12802 2021-03-12T02:24:26.424 INFO:teuthology.task.ansible.out:9", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice docker.socket", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-03-12 02:20:43 UTC", "StateChangeTimestampMonotonic": "40242558", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "Su 2021-03-12T02:24:26.426 INFO:teuthology.task.ansible.out:ccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-12T02:24:26.491 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-03-12T02:24:26.492 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-03-12T02:24:26.494 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi077.front.sepia.ceph.com : ok=127 changed=35 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 smithi114.front.sepia.ceph.com : ok=127 changed=35 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 Friday 12 March 2021 02:24:26 +0000 (0:00:03.328) 0:03:06.782 ********** =============================================================================== 2021-03-12T02:24:26.496 INFO:teuthology.task.ansible.out:Create all admin users with sudo access. ------------------------------- 21.57s users : Update authorized_keys using the keys repo --------------------- 10.10s container-host : Install registries-conf-ctl ---------------------------- 3.49s container-host : Restart docker service --------------------------------- 3.33s testnode : Update apt cache. -------------------------------------------- 3.00s testnode : Install apt keys --------------------------------------------- 2.20s Gathering Facts --------------------------------------------------------- 1.86s testnode : Zap all non-root disks --------------------------------------- 1.36s users : Clone the keys repo --------------------------------------------- 1.22s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.17s testnode : Ensure packages are not present. ----------------------------- 1.16s testnode : Install python-apt ------------------------------------------- 1.06s testnode : Install packages --------------------------------------------- 1.05s testnode : Make sure apt dependencies are installed --------------------- 0.98s testnode : Upgrade packages --------------------------------------------- 0.98s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.94s container-host : Install container packages ----------------------------- 0.94s testnode : Ensure perl-doc is installed on apt systems. ----------------- 0.93s testnode : Install python-pip on apt based systems. --------------------- 0.93s testnode : Install ntp package on apt based systems. -------------------- 0.91s 2021-03-12T02:24:26.927 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi114.front.sepia.ceph.com')] 2021-03-12T02:24:26.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-12T02:24:26.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-12T02:24:27.045 DEBUG:teuthology.orchestra.run.smithi077:> true 2021-03-12T02:24:27.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-12T02:24:27.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-12T02:24:27.319 DEBUG:teuthology.orchestra.run.smithi114:> true 2021-03-12T02:24:27.519 INFO:teuthology.run_tasks:Running task clock... 2021-03-12T02:24:27.531 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-03-12T02:24:27.532 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-12T02:24:27.533 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-12T02:24:27.599 INFO:teuthology.orchestra.run.smithi077.stdout:12 Mar 02:24:27 ntpd[9283]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-03-12T02:24:27.599 INFO:teuthology.orchestra.run.smithi077.stdout:12 Mar 02:24:27 ntpd[9283]: Command line: ntpd -gq 2021-03-12T02:24:27.600 INFO:teuthology.orchestra.run.smithi077.stdout:12 Mar 02:24:27 ntpd[9283]: proto: precision = 0.123 usec (-23) 2021-03-12T02:24:27.600 INFO:teuthology.orchestra.run.smithi077.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-12T02:24:27.601 INFO:teuthology.orchestra.run.smithi077.stdout:12 Mar 02:24:27 ntpd[9283]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-12T02:24:27.601 INFO:teuthology.orchestra.run.smithi077.stderr:restrict ::: KOD does nothing without LIMITED. 2021-03-12T02:24:27.602 INFO:teuthology.orchestra.run.smithi077.stdout:12 Mar 02:24:27 ntpd[9283]: restrict ::: KOD does nothing without LIMITED. 2021-03-12T02:24:27.603 INFO:teuthology.orchestra.run.smithi077.stdout:12 Mar 02:24:27 ntpd[9283]: Listen and drop on 0 v6wildcard [::]:123 2021-03-12T02:24:27.603 INFO:teuthology.orchestra.run.smithi077.stdout:12 Mar 02:24:27 ntpd[9283]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-03-12T02:24:27.603 INFO:teuthology.orchestra.run.smithi077.stdout:12 Mar 02:24:27 ntpd[9283]: Listen normally on 2 lo 127.0.0.1:123 2021-03-12T02:24:27.603 INFO:teuthology.orchestra.run.smithi077.stdout:12 Mar 02:24:27 ntpd[9283]: Listen normally on 3 enp3s0f1 172.21.15.77:123 2021-03-12T02:24:27.604 INFO:teuthology.orchestra.run.smithi077.stdout:12 Mar 02:24:27 ntpd[9283]: Listen normally on 4 lo [::1]:123 2021-03-12T02:24:27.604 INFO:teuthology.orchestra.run.smithi077.stdout:12 Mar 02:24:27 ntpd[9283]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d3d3%5]:123 2021-03-12T02:24:27.604 INFO:teuthology.orchestra.run.smithi077.stdout:12 Mar 02:24:27 ntpd[9283]: Listening on routing socket on fd #22 for interface updates 2021-03-12T02:24:28.599 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:28 ntpd[9283]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:28.600 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:28 ntpd[9283]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:28.600 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:28 ntpd[9283]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:28.600 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:28 ntpd[9283]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:29.600 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:29 ntpd[9283]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:29.600 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:29 ntpd[9283]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:29.600 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:29 ntpd[9283]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:29.601 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:29 ntpd[9283]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:30.599 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:30 ntpd[9283]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:30.600 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:30 ntpd[9283]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:30.600 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:30 ntpd[9283]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:30.600 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:30 ntpd[9283]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:31.600 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:31 ntpd[9283]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:31.600 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:31 ntpd[9283]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:31.600 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:31 ntpd[9283]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:31.601 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:31 ntpd[9283]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:31.601 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:31 ntpd[9283]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:31.601 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:31 ntpd[9283]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:31.601 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:31 ntpd[9283]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:31.602 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:31 ntpd[9283]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:31.602 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:31 ntpd[9283]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:31.602 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:31 ntpd[9283]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:31.602 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:31 ntpd[9283]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:31.602 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:31 ntpd[9283]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:32.600 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:32 ntpd[9283]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:32.600 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:32 ntpd[9283]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:32.600 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:32 ntpd[9283]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:32.600 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:32 ntpd[9283]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:33.601 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:33 ntpd[9283]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:33.602 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:33 ntpd[9283]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:33.602 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:33 ntpd[9283]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:33.603 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:33 ntpd[9283]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:33.603 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:33 ntpd[9283]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:33.603 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:33 ntpd[9283]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:33.605 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:33 ntpd[9283]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:33.605 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:33 ntpd[9283]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:33.605 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:33 ntpd[9283]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:33.606 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:33 ntpd[9283]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:33.606 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:33 ntpd[9283]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:33.606 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:33 ntpd[9283]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:34.600 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:34 ntpd[9283]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:34.601 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:34 ntpd[9283]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:34.601 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:34 ntpd[9283]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:34.602 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:34 ntpd[9283]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:34.606 INFO:teuthology.orchestra.run.smithi077.stdout:12 Mar 02:24:34 ntpd[9283]: ntpd: time slew +0.003287 s 2021-03-12T02:24:34.606 INFO:teuthology.orchestra.run.smithi077.stdout:ntpd: time slew +0.003287s 2021-03-12T02:24:34.608 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:34 ntpd[9283]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-03-12T02:24:34.608 INFO:teuthology.orchestra.run.smithi077.stderr:12 Mar 02:24:34 ntpd[9283]: can't open /var/log/ntpstats/loopstats.20210312: Permission denied 2021-03-12T02:24:34.788 INFO:teuthology.orchestra.run.smithi077.stdout: remote refid st t when poll reach delay offset jitter 2021-03-12T02:24:34.788 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================== 2021-03-12T02:24:34.789 INFO:teuthology.orchestra.run.smithi077.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-12T02:24:34.789 INFO:teuthology.orchestra.run.smithi077.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-12T02:24:34.789 INFO:teuthology.orchestra.run.smithi077.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-12T02:24:34.789 INFO:teuthology.orchestra.run.smithi077.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-12T02:24:34.790 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-12T02:24:34.791 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-12T02:24:34.850 INFO:teuthology.orchestra.run.smithi114.stdout:12 Mar 02:24:34 ntpd[9333]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-03-12T02:24:34.850 INFO:teuthology.orchestra.run.smithi114.stdout:12 Mar 02:24:34 ntpd[9333]: Command line: ntpd -gq 2021-03-12T02:24:34.851 INFO:teuthology.orchestra.run.smithi114.stdout:12 Mar 02:24:34 ntpd[9333]: proto: precision = 0.123 usec (-23) 2021-03-12T02:24:34.851 INFO:teuthology.orchestra.run.smithi114.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-12T02:24:34.852 INFO:teuthology.orchestra.run.smithi114.stdout:12 Mar 02:24:34 ntpd[9333]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-12T02:24:34.852 INFO:teuthology.orchestra.run.smithi114.stderr:restrict ::: KOD does nothing without LIMITED. 2021-03-12T02:24:34.853 INFO:teuthology.orchestra.run.smithi114.stdout:12 Mar 02:24:34 ntpd[9333]: restrict ::: KOD does nothing without LIMITED. 2021-03-12T02:24:34.854 INFO:teuthology.orchestra.run.smithi114.stdout:12 Mar 02:24:34 ntpd[9333]: Listen and drop on 0 v6wildcard [::]:123 2021-03-12T02:24:34.854 INFO:teuthology.orchestra.run.smithi114.stdout:12 Mar 02:24:34 ntpd[9333]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-03-12T02:24:34.856 INFO:teuthology.orchestra.run.smithi114.stdout:12 Mar 02:24:34 ntpd[9333]: Listen normally on 2 lo 127.0.0.1:123 2021-03-12T02:24:34.856 INFO:teuthology.orchestra.run.smithi114.stdout:12 Mar 02:24:34 ntpd[9333]: Listen normally on 3 enp3s0f1 172.21.15.114:123 2021-03-12T02:24:34.856 INFO:teuthology.orchestra.run.smithi114.stdout:12 Mar 02:24:34 ntpd[9333]: Listen normally on 4 lo [::1]:123 2021-03-12T02:24:34.857 INFO:teuthology.orchestra.run.smithi114.stdout:12 Mar 02:24:34 ntpd[9333]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cb63%5]:123 2021-03-12T02:24:34.857 INFO:teuthology.orchestra.run.smithi114.stdout:12 Mar 02:24:34 ntpd[9333]: Listening on routing socket on fd #22 for interface updates 2021-03-12T02:24:35.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:35 ntpd[9333]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:35.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:35 ntpd[9333]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:35.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:35 ntpd[9333]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:35.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:35 ntpd[9333]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:35.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:35 ntpd[9333]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:35.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:35 ntpd[9333]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:35.853 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:35 ntpd[9333]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:35.853 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:35 ntpd[9333]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:36.850 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:36 ntpd[9333]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:36.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:36 ntpd[9333]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:36.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:36 ntpd[9333]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:36.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:36 ntpd[9333]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:37.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:37 ntpd[9333]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:37.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:37 ntpd[9333]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:37.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:37 ntpd[9333]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:37.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:37 ntpd[9333]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:37.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:37 ntpd[9333]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:37.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:37 ntpd[9333]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:37.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:37 ntpd[9333]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:37.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:37 ntpd[9333]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:38.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:38 ntpd[9333]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:38.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:38 ntpd[9333]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:38.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:38 ntpd[9333]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:38.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:38 ntpd[9333]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:38.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:38 ntpd[9333]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:38.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:38 ntpd[9333]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:38.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:38 ntpd[9333]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:38.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:38 ntpd[9333]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:39.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:39 ntpd[9333]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:39.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:39 ntpd[9333]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:39.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:39 ntpd[9333]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:39.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:39 ntpd[9333]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:39.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:39 ntpd[9333]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:39.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:39 ntpd[9333]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:39.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:39 ntpd[9333]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:39.853 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:39 ntpd[9333]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:40.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:40 ntpd[9333]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:40.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:40 ntpd[9333]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:40.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:40 ntpd[9333]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:40.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:40 ntpd[9333]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:40.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:40 ntpd[9333]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:40.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:40 ntpd[9333]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:40.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:40 ntpd[9333]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:40.852 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:40 ntpd[9333]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:41.850 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:41 ntpd[9333]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-12T02:24:41.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:41 ntpd[9333]: can't open /var/log/ntpstats/rawstats.20210312: Permission denied 2021-03-12T02:24:41.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:41 ntpd[9333]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-12T02:24:41.851 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:41 ntpd[9333]: can't open /var/log/ntpstats/peerstats.20210312: Permission denied 2021-03-12T02:24:41.852 INFO:teuthology.orchestra.run.smithi114.stdout:12 Mar 02:24:41 ntpd[9333]: ntpd: time slew +0.001359 s 2021-03-12T02:24:41.852 INFO:teuthology.orchestra.run.smithi114.stdout:ntpd: time slew +0.001359s 2021-03-12T02:24:41.853 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:41 ntpd[9333]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-03-12T02:24:41.853 INFO:teuthology.orchestra.run.smithi114.stderr:12 Mar 02:24:41 ntpd[9333]: can't open /var/log/ntpstats/loopstats.20210312: Permission denied 2021-03-12T02:24:41.981 INFO:teuthology.orchestra.run.smithi114.stdout: remote refid st t when poll reach delay offset jitter 2021-03-12T02:24:41.981 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================== 2021-03-12T02:24:41.982 INFO:teuthology.orchestra.run.smithi114.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-12T02:24:41.982 INFO:teuthology.orchestra.run.smithi114.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-12T02:24:41.982 INFO:teuthology.orchestra.run.smithi114.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-12T02:24:41.982 INFO:teuthology.orchestra.run.smithi114.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-12T02:24:41.983 INFO:teuthology.run_tasks:Running task install... 2021-03-12T02:24:41.991 DEBUG:teuthology.task.install:project ceph 2021-03-12T02:24:41.992 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb'}} 2021-03-12T02:24:41.992 DEBUG:teuthology.task.install:config {'sha1': '3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb'} 2021-03-12T02:24:41.992 INFO:teuthology.task.install:Using flavor: basic 2021-03-12T02:24:42.004 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-03-12T02:24:42.004 INFO:teuthology.task.install:extra packages: [] 2021-03-12T02:24:42.005 DEBUG:teuthology.orchestra.run.smithi077:> sudo lsb_release -is 2021-03-12T02:24:42.059 INFO:teuthology.orchestra.run.smithi077.stdout:Ubuntu 2021-03-12T02:24:42.065 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-12T02:24:42.065 DEBUG:teuthology.orchestra.run.smithi114:> sudo lsb_release -is 2021-03-12T02:24:42.066 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-key list | grep Ceph 2021-03-12T02:24:42.120 INFO:teuthology.orchestra.run.smithi114.stdout:Ubuntu 2021-03-12T02:24:42.126 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-12T02:24:42.126 DEBUG:teuthology.orchestra.run.smithi114:> sudo apt-key list | grep Ceph 2021-03-12T02:24:42.147 INFO:teuthology.orchestra.run.smithi077.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-12T02:24:42.203 INFO:teuthology.orchestra.run.smithi077.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-03-12T02:24:42.204 INFO:teuthology.orchestra.run.smithi077.stdout:uid [ unknown] Ceph.com (release key) 2021-03-12T02:24:42.205 DEBUG:teuthology.orchestra.run.smithi077:> sudo lsb_release -is 2021-03-12T02:24:42.218 INFO:teuthology.orchestra.run.smithi114.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-12T02:24:42.256 INFO:teuthology.orchestra.run.smithi077.stdout:Ubuntu 2021-03-12T02:24:42.262 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-12T02:24:42.263 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-03-12T02:24:42.263 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb 2021-03-12T02:24:42.273 INFO:teuthology.orchestra.run.smithi114.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-03-12T02:24:42.273 INFO:teuthology.orchestra.run.smithi114.stdout:uid [ unknown] Ceph.com (release key) 2021-03-12T02:24:42.274 DEBUG:teuthology.orchestra.run.smithi114:> sudo lsb_release -is 2021-03-12T02:24:42.326 INFO:teuthology.orchestra.run.smithi114.stdout:Ubuntu 2021-03-12T02:24:42.332 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-12T02:24:42.332 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-03-12T02:24:42.332 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb 2021-03-12T02:24:42.412 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default/ 2021-03-12T02:24:42.412 INFO:teuthology.task.install.deb:Package version is 15.2.9-120-g3cd904d7-1bionic 2021-03-12T02:24:42.466 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default/ 2021-03-12T02:24:42.467 INFO:teuthology.task.install.deb:Package version is 15.2.9-120-g3cd904d7-1bionic 2021-03-12T02:24:42.540 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2021-03-12T02:24:42.540 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-03-12T02:24:42.555 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-get update 2021-03-12T02:24:42.582 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2021-03-12T02:24:42.582 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-03-12T02:24:42.603 DEBUG:teuthology.orchestra.run.smithi114:> sudo apt-get update 2021-03-12T02:24:42.790 INFO:teuthology.orchestra.run.smithi077.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic InRelease 2021-03-12T02:24:42.810 INFO:teuthology.orchestra.run.smithi077.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-03-12T02:24:42.832 INFO:teuthology.orchestra.run.smithi077.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic Release.gpg 2021-03-12T02:24:42.841 INFO:teuthology.orchestra.run.smithi114.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic InRelease 2021-03-12T02:24:42.860 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-12T02:24:42.861 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:5 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-03-12T02:24:42.861 INFO:teuthology.orchestra.run.smithi077.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.8 kB] 2021-03-12T02:24:42.862 INFO:teuthology.orchestra.run.smithi114.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-03-12T02:24:42.884 INFO:teuthology.orchestra.run.smithi114.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic Release.gpg 2021-03-12T02:24:42.893 INFO:teuthology.orchestra.run.smithi077.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,090 B] 2021-03-12T02:24:42.918 INFO:teuthology.orchestra.run.smithi114.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-12T02:24:42.918 INFO:teuthology.orchestra.run.smithi114.stdout:Hit:5 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-03-12T02:24:42.918 INFO:teuthology.orchestra.run.smithi114.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.8 kB] 2021-03-12T02:24:42.930 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-03-12T02:24:42.944 INFO:teuthology.orchestra.run.smithi114.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,090 B] 2021-03-12T02:24:42.977 INFO:teuthology.orchestra.run.smithi114.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-03-12T02:24:43.004 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-03-12T02:24:43.062 INFO:teuthology.orchestra.run.smithi114.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-03-12T02:24:43.466 INFO:teuthology.orchestra.run.smithi077.stdout:Fetched 26.6 kB in 1s (37.6 kB/s) 2021-03-12T02:24:43.518 INFO:teuthology.orchestra.run.smithi114.stdout:Fetched 26.6 kB in 1s (37.6 kB/s) 2021-03-12T02:24:44.706 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2021-03-12T02:24:44.753 DEBUG:teuthology.orchestra.run.smithi077:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.9-120-g3cd904d7-1bionic cephadm=15.2.9-120-g3cd904d7-1bionic ceph-mds=15.2.9-120-g3cd904d7-1bionic ceph-mgr=15.2.9-120-g3cd904d7-1bionic ceph-common=15.2.9-120-g3cd904d7-1bionic ceph-fuse=15.2.9-120-g3cd904d7-1bionic ceph-test=15.2.9-120-g3cd904d7-1bionic radosgw=15.2.9-120-g3cd904d7-1bionic python3-rados=15.2.9-120-g3cd904d7-1bionic python3-rgw=15.2.9-120-g3cd904d7-1bionic python3-cephfs=15.2.9-120-g3cd904d7-1bionic python3-rbd=15.2.9-120-g3cd904d7-1bionic libcephfs2=15.2.9-120-g3cd904d7-1bionic libcephfs-dev=15.2.9-120-g3cd904d7-1bionic librados2=15.2.9-120-g3cd904d7-1bionic librbd1=15.2.9-120-g3cd904d7-1bionic rbd-fuse=15.2.9-120-g3cd904d7-1bionic 2021-03-12T02:24:44.754 INFO:teuthology.orchestra.run.smithi114.stdout:Reading package lists... 2021-03-12T02:24:44.801 DEBUG:teuthology.orchestra.run.smithi114:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.9-120-g3cd904d7-1bionic cephadm=15.2.9-120-g3cd904d7-1bionic ceph-mds=15.2.9-120-g3cd904d7-1bionic ceph-mgr=15.2.9-120-g3cd904d7-1bionic ceph-common=15.2.9-120-g3cd904d7-1bionic ceph-fuse=15.2.9-120-g3cd904d7-1bionic ceph-test=15.2.9-120-g3cd904d7-1bionic radosgw=15.2.9-120-g3cd904d7-1bionic python3-rados=15.2.9-120-g3cd904d7-1bionic python3-rgw=15.2.9-120-g3cd904d7-1bionic python3-cephfs=15.2.9-120-g3cd904d7-1bionic python3-rbd=15.2.9-120-g3cd904d7-1bionic libcephfs2=15.2.9-120-g3cd904d7-1bionic libcephfs-dev=15.2.9-120-g3cd904d7-1bionic librados2=15.2.9-120-g3cd904d7-1bionic librbd1=15.2.9-120-g3cd904d7-1bionic rbd-fuse=15.2.9-120-g3cd904d7-1bionic 2021-03-12T02:24:44.844 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2021-03-12T02:24:44.884 INFO:teuthology.orchestra.run.smithi114.stdout:Reading package lists... 2021-03-12T02:24:45.103 INFO:teuthology.orchestra.run.smithi077.stdout:Building dependency tree... 2021-03-12T02:24:45.104 INFO:teuthology.orchestra.run.smithi077.stdout:Reading state information... 2021-03-12T02:24:45.137 INFO:teuthology.orchestra.run.smithi114.stdout:Building dependency tree... 2021-03-12T02:24:45.138 INFO:teuthology.orchestra.run.smithi114.stdout:Reading state information... 2021-03-12T02:24:45.269 INFO:teuthology.orchestra.run.smithi077.stdout:The following additional packages will be installed: 2021-03-12T02:24:45.269 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2021-03-12T02:24:45.270 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2021-03-12T02:24:45.271 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2021-03-12T02:24:45.271 INFO:teuthology.orchestra.run.smithi077.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-03-12T02:24:45.271 INFO:teuthology.orchestra.run.smithi077.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-03-12T02:24:45.272 INFO:teuthology.orchestra.run.smithi077.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-03-12T02:24:45.272 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-12T02:24:45.272 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-03-12T02:24:45.272 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-03-12T02:24:45.272 INFO:teuthology.orchestra.run.smithi077.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-03-12T02:24:45.273 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-03-12T02:24:45.273 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-03-12T02:24:45.273 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-03-12T02:24:45.273 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-03-12T02:24:45.274 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-03-12T02:24:45.274 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-03-12T02:24:45.274 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-03-12T02:24:45.274 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-03-12T02:24:45.274 INFO:teuthology.orchestra.run.smithi077.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-03-12T02:24:45.275 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-03-12T02:24:45.275 INFO:teuthology.orchestra.run.smithi077.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-03-12T02:24:45.275 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-03-12T02:24:45.275 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-03-12T02:24:45.277 INFO:teuthology.orchestra.run.smithi077.stdout:Suggested packages: 2021-03-12T02:24:45.277 INFO:teuthology.orchestra.run.smithi077.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-03-12T02:24:45.277 INFO:teuthology.orchestra.run.smithi077.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-03-12T02:24:45.278 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-03-12T02:24:45.278 INFO:teuthology.orchestra.run.smithi077.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-03-12T02:24:45.279 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-03-12T02:24:45.279 INFO:teuthology.orchestra.run.smithi077.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-03-12T02:24:45.279 INFO:teuthology.orchestra.run.smithi077.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-03-12T02:24:45.280 INFO:teuthology.orchestra.run.smithi077.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-03-12T02:24:45.280 INFO:teuthology.orchestra.run.smithi077.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-03-12T02:24:45.280 INFO:teuthology.orchestra.run.smithi077.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-03-12T02:24:45.280 INFO:teuthology.orchestra.run.smithi077.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-03-12T02:24:45.281 INFO:teuthology.orchestra.run.smithi077.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-03-12T02:24:45.281 INFO:teuthology.orchestra.run.smithi077.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-03-12T02:24:45.282 INFO:teuthology.orchestra.run.smithi077.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-03-12T02:24:45.282 INFO:teuthology.orchestra.run.smithi077.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-03-12T02:24:45.304 INFO:teuthology.orchestra.run.smithi114.stdout:The following additional packages will be installed: 2021-03-12T02:24:45.304 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2021-03-12T02:24:45.304 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2021-03-12T02:24:45.305 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2021-03-12T02:24:45.306 INFO:teuthology.orchestra.run.smithi114.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-03-12T02:24:45.306 INFO:teuthology.orchestra.run.smithi114.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-03-12T02:24:45.306 INFO:teuthology.orchestra.run.smithi114.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-03-12T02:24:45.306 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-12T02:24:45.307 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-03-12T02:24:45.307 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-03-12T02:24:45.307 INFO:teuthology.orchestra.run.smithi114.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-03-12T02:24:45.307 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-03-12T02:24:45.308 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-03-12T02:24:45.308 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-03-12T02:24:45.308 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-03-12T02:24:45.308 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-03-12T02:24:45.308 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-03-12T02:24:45.309 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-03-12T02:24:45.309 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-03-12T02:24:45.309 INFO:teuthology.orchestra.run.smithi114.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-03-12T02:24:45.309 INFO:teuthology.orchestra.run.smithi114.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-03-12T02:24:45.310 INFO:teuthology.orchestra.run.smithi114.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-03-12T02:24:45.310 INFO:teuthology.orchestra.run.smithi114.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-03-12T02:24:45.310 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-03-12T02:24:45.311 INFO:teuthology.orchestra.run.smithi114.stdout:Suggested packages: 2021-03-12T02:24:45.311 INFO:teuthology.orchestra.run.smithi114.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-03-12T02:24:45.311 INFO:teuthology.orchestra.run.smithi114.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-03-12T02:24:45.311 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-03-12T02:24:45.312 INFO:teuthology.orchestra.run.smithi114.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-03-12T02:24:45.312 INFO:teuthology.orchestra.run.smithi114.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-03-12T02:24:45.312 INFO:teuthology.orchestra.run.smithi114.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-03-12T02:24:45.312 INFO:teuthology.orchestra.run.smithi114.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-03-12T02:24:45.312 INFO:teuthology.orchestra.run.smithi114.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-03-12T02:24:45.313 INFO:teuthology.orchestra.run.smithi114.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-03-12T02:24:45.313 INFO:teuthology.orchestra.run.smithi114.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-03-12T02:24:45.313 INFO:teuthology.orchestra.run.smithi114.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-03-12T02:24:45.313 INFO:teuthology.orchestra.run.smithi114.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-03-12T02:24:45.313 INFO:teuthology.orchestra.run.smithi114.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-03-12T02:24:45.314 INFO:teuthology.orchestra.run.smithi114.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-03-12T02:24:45.314 INFO:teuthology.orchestra.run.smithi114.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-03-12T02:24:45.376 INFO:teuthology.orchestra.run.smithi077.stdout:The following NEW packages will be installed: 2021-03-12T02:24:45.376 INFO:teuthology.orchestra.run.smithi077.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-03-12T02:24:45.377 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2021-03-12T02:24:45.377 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2021-03-12T02:24:45.377 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2021-03-12T02:24:45.379 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-03-12T02:24:45.379 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-03-12T02:24:45.379 INFO:teuthology.orchestra.run.smithi077.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-03-12T02:24:45.379 INFO:teuthology.orchestra.run.smithi077.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-12T02:24:45.380 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-03-12T02:24:45.380 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-03-12T02:24:45.380 INFO:teuthology.orchestra.run.smithi077.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-03-12T02:24:45.380 INFO:teuthology.orchestra.run.smithi077.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-03-12T02:24:45.380 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-03-12T02:24:45.381 INFO:teuthology.orchestra.run.smithi077.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-03-12T02:24:45.381 INFO:teuthology.orchestra.run.smithi077.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-03-12T02:24:45.381 INFO:teuthology.orchestra.run.smithi077.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-03-12T02:24:45.381 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-03-12T02:24:45.381 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-03-12T02:24:45.382 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-03-12T02:24:45.382 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-03-12T02:24:45.382 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-03-12T02:24:45.382 INFO:teuthology.orchestra.run.smithi077.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-03-12T02:24:45.383 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-03-12T02:24:45.383 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-03-12T02:24:45.383 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-03-12T02:24:45.383 INFO:teuthology.orchestra.run.smithi077.stdout:The following packages will be upgraded: 2021-03-12T02:24:45.384 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 librbd1 2021-03-12T02:24:45.408 INFO:teuthology.orchestra.run.smithi114.stdout:The following NEW packages will be installed: 2021-03-12T02:24:45.408 INFO:teuthology.orchestra.run.smithi114.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-03-12T02:24:45.409 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2021-03-12T02:24:45.409 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2021-03-12T02:24:45.409 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2021-03-12T02:24:45.410 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-03-12T02:24:45.411 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-03-12T02:24:45.411 INFO:teuthology.orchestra.run.smithi114.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-03-12T02:24:45.411 INFO:teuthology.orchestra.run.smithi114.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-12T02:24:45.411 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-03-12T02:24:45.412 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-03-12T02:24:45.412 INFO:teuthology.orchestra.run.smithi114.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-03-12T02:24:45.412 INFO:teuthology.orchestra.run.smithi114.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-03-12T02:24:45.412 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-03-12T02:24:45.413 INFO:teuthology.orchestra.run.smithi114.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-03-12T02:24:45.413 INFO:teuthology.orchestra.run.smithi114.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-03-12T02:24:45.413 INFO:teuthology.orchestra.run.smithi114.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-03-12T02:24:45.413 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-03-12T02:24:45.413 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-03-12T02:24:45.414 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-03-12T02:24:45.414 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-03-12T02:24:45.414 INFO:teuthology.orchestra.run.smithi114.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-03-12T02:24:45.414 INFO:teuthology.orchestra.run.smithi114.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-03-12T02:24:45.415 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-03-12T02:24:45.415 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-03-12T02:24:45.415 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-03-12T02:24:45.416 INFO:teuthology.orchestra.run.smithi114.stdout:The following packages will be upgraded: 2021-03-12T02:24:45.416 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 librbd1 2021-03-12T02:24:45.529 INFO:teuthology.orchestra.run.smithi077.stdout:2 upgraded, 111 newly installed, 0 to remove and 17 not upgraded. 2021-03-12T02:24:45.529 INFO:teuthology.orchestra.run.smithi077.stdout:Need to get 156 MB of archives. 2021-03-12T02:24:45.529 INFO:teuthology.orchestra.run.smithi077.stdout:After this operation, 725 MB of additional disk space will be used. 2021-03-12T02:24:45.529 INFO:teuthology.orchestra.run.smithi077.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.2.9-120-g3cd904d7-1bionic [1,638 kB] 2021-03-12T02:24:45.561 INFO:teuthology.orchestra.run.smithi114.stdout:2 upgraded, 111 newly installed, 0 to remove and 17 not upgraded. 2021-03-12T02:24:45.561 INFO:teuthology.orchestra.run.smithi114.stdout:Need to get 156 MB of archives. 2021-03-12T02:24:45.561 INFO:teuthology.orchestra.run.smithi114.stdout:After this operation, 725 MB of additional disk space will be used. 2021-03-12T02:24:45.562 INFO:teuthology.orchestra.run.smithi114.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.2.9-120-g3cd904d7-1bionic [1,638 kB] 2021-03-12T02:24:45.566 INFO:teuthology.orchestra.run.smithi077.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-03-12T02:24:45.601 INFO:teuthology.orchestra.run.smithi114.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-03-12T02:24:45.646 INFO:teuthology.orchestra.run.smithi077.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-03-12T02:24:45.681 INFO:teuthology.orchestra.run.smithi114.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-03-12T02:24:45.710 INFO:teuthology.orchestra.run.smithi077.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.2.9-120-g3cd904d7-1bionic [3,061 kB] 2021-03-12T02:24:45.733 INFO:teuthology.orchestra.run.smithi114.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.2.9-120-g3cd904d7-1bionic [3,061 kB] 2021-03-12T02:24:45.869 INFO:teuthology.orchestra.run.smithi077.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.2.9-120-g3cd904d7-1bionic [436 kB] 2021-03-12T02:24:45.897 INFO:teuthology.orchestra.run.smithi077.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-03-12T02:24:45.905 INFO:teuthology.orchestra.run.smithi114.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.2.9-120-g3cd904d7-1bionic [436 kB] 2021-03-12T02:24:45.910 INFO:teuthology.orchestra.run.smithi077.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-03-12T02:24:45.920 INFO:teuthology.orchestra.run.smithi077.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.2.9-120-g3cd904d7-1bionic [346 kB] 2021-03-12T02:24:45.931 INFO:teuthology.orchestra.run.smithi077.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-03-12T02:24:45.931 INFO:teuthology.orchestra.run.smithi077.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.2.9-120-g3cd904d7-1bionic [20.7 kB] 2021-03-12T02:24:45.931 INFO:teuthology.orchestra.run.smithi077.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.2.9-120-g3cd904d7-1bionic [133 kB] 2021-03-12T02:24:45.936 INFO:teuthology.orchestra.run.smithi077.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 15.2.9-120-g3cd904d7-1bionic [29.1 kB] 2021-03-12T02:24:45.937 INFO:teuthology.orchestra.run.smithi077.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 15.2.9-120-g3cd904d7-1bionic [326 kB] 2021-03-12T02:24:45.938 INFO:teuthology.orchestra.run.smithi114.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-03-12T02:24:45.942 INFO:teuthology.orchestra.run.smithi114.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-03-12T02:24:45.953 INFO:teuthology.orchestra.run.smithi114.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-03-12T02:24:45.957 INFO:teuthology.orchestra.run.smithi114.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.2.9-120-g3cd904d7-1bionic [346 kB] 2021-03-12T02:24:45.969 INFO:teuthology.orchestra.run.smithi114.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.2.9-120-g3cd904d7-1bionic [20.7 kB] 2021-03-12T02:24:45.969 INFO:teuthology.orchestra.run.smithi114.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.2.9-120-g3cd904d7-1bionic [133 kB] 2021-03-12T02:24:45.974 INFO:teuthology.orchestra.run.smithi114.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 15.2.9-120-g3cd904d7-1bionic [29.1 kB] 2021-03-12T02:24:45.975 INFO:teuthology.orchestra.run.smithi114.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 15.2.9-120-g3cd904d7-1bionic [326 kB] 2021-03-12T02:24:45.986 INFO:teuthology.orchestra.run.smithi077.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.2.9-120-g3cd904d7-1bionic [2,905 kB] 2021-03-12T02:24:46.026 INFO:teuthology.orchestra.run.smithi114.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.2.9-120-g3cd904d7-1bionic [2,905 kB] 2021-03-12T02:24:46.028 INFO:teuthology.orchestra.run.smithi077.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-03-12T02:24:46.032 INFO:teuthology.orchestra.run.smithi077.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-03-12T02:24:46.056 INFO:teuthology.orchestra.run.smithi114.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-03-12T02:24:46.056 INFO:teuthology.orchestra.run.smithi077.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-03-12T02:24:46.059 INFO:teuthology.orchestra.run.smithi114.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-03-12T02:24:46.064 INFO:teuthology.orchestra.run.smithi077.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-03-12T02:24:46.071 INFO:teuthology.orchestra.run.smithi077.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-03-12T02:24:46.079 INFO:teuthology.orchestra.run.smithi077.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-03-12T02:24:46.090 INFO:teuthology.orchestra.run.smithi077.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-03-12T02:24:46.095 INFO:teuthology.orchestra.run.smithi114.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-03-12T02:24:46.101 INFO:teuthology.orchestra.run.smithi114.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-03-12T02:24:46.110 INFO:teuthology.orchestra.run.smithi114.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-03-12T02:24:46.117 INFO:teuthology.orchestra.run.smithi077.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-03-12T02:24:46.122 INFO:teuthology.orchestra.run.smithi114.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-03-12T02:24:46.123 INFO:teuthology.orchestra.run.smithi077.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-03-12T02:24:46.131 INFO:teuthology.orchestra.run.smithi114.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-03-12T02:24:46.135 INFO:teuthology.orchestra.run.smithi077.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-03-12T02:24:46.148 INFO:teuthology.orchestra.run.smithi077.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-03-12T02:24:46.149 INFO:teuthology.orchestra.run.smithi077.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-03-12T02:24:46.160 INFO:teuthology.orchestra.run.smithi114.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-03-12T02:24:46.164 INFO:teuthology.orchestra.run.smithi114.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-03-12T02:24:46.178 INFO:teuthology.orchestra.run.smithi114.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-03-12T02:24:46.188 INFO:teuthology.orchestra.run.smithi114.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-03-12T02:24:46.190 INFO:teuthology.orchestra.run.smithi114.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-03-12T02:24:46.214 INFO:teuthology.orchestra.run.smithi077.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-03-12T02:24:46.215 INFO:teuthology.orchestra.run.smithi077.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-03-12T02:24:46.217 INFO:teuthology.orchestra.run.smithi077.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-03-12T02:24:46.217 INFO:teuthology.orchestra.run.smithi077.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-03-12T02:24:46.218 INFO:teuthology.orchestra.run.smithi077.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-03-12T02:24:46.221 INFO:teuthology.orchestra.run.smithi077.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-03-12T02:24:46.221 INFO:teuthology.orchestra.run.smithi077.stdout:Get:33 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 15.2.9-120-g3cd904d7-1bionic [99.3 kB] 2021-03-12T02:24:46.223 INFO:teuthology.orchestra.run.smithi077.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-03-12T02:24:46.223 INFO:teuthology.orchestra.run.smithi077.stdout:Get:35 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.2.9-120-g3cd904d7-1bionic [338 kB] 2021-03-12T02:24:46.232 INFO:teuthology.orchestra.run.smithi077.stdout:Get:36 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.2.9-120-g3cd904d7-1bionic [18.8 MB] 2021-03-12T02:24:46.249 INFO:teuthology.orchestra.run.smithi114.stdout:Get:27 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 15.2.9-120-g3cd904d7-1bionic [99.3 kB] 2021-03-12T02:24:46.250 INFO:teuthology.orchestra.run.smithi114.stdout:Get:28 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.2.9-120-g3cd904d7-1bionic [338 kB] 2021-03-12T02:24:46.253 INFO:teuthology.orchestra.run.smithi114.stdout:Get:29 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.2.9-120-g3cd904d7-1bionic [18.8 MB] 2021-03-12T02:24:46.257 INFO:teuthology.orchestra.run.smithi114.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-03-12T02:24:46.258 INFO:teuthology.orchestra.run.smithi114.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-03-12T02:24:46.260 INFO:teuthology.orchestra.run.smithi114.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-03-12T02:24:46.265 INFO:teuthology.orchestra.run.smithi114.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-03-12T02:24:46.265 INFO:teuthology.orchestra.run.smithi114.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-03-12T02:24:46.265 INFO:teuthology.orchestra.run.smithi114.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-03-12T02:24:46.266 INFO:teuthology.orchestra.run.smithi114.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-03-12T02:24:46.293 INFO:teuthology.orchestra.run.smithi077.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-03-12T02:24:46.493 INFO:teuthology.orchestra.run.smithi114.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-03-12T02:24:46.493 INFO:teuthology.orchestra.run.smithi114.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-03-12T02:24:46.493 INFO:teuthology.orchestra.run.smithi114.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-03-12T02:24:46.494 INFO:teuthology.orchestra.run.smithi114.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-03-12T02:24:46.494 INFO:teuthology.orchestra.run.smithi114.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-03-12T02:24:46.494 INFO:teuthology.orchestra.run.smithi114.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-03-12T02:24:46.494 INFO:teuthology.orchestra.run.smithi114.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-03-12T02:24:46.495 INFO:teuthology.orchestra.run.smithi114.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-03-12T02:24:46.495 INFO:teuthology.orchestra.run.smithi114.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-03-12T02:24:46.495 INFO:teuthology.orchestra.run.smithi114.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-03-12T02:24:46.495 INFO:teuthology.orchestra.run.smithi077.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-03-12T02:24:46.495 INFO:teuthology.orchestra.run.smithi077.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-03-12T02:24:46.496 INFO:teuthology.orchestra.run.smithi077.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-03-12T02:24:46.496 INFO:teuthology.orchestra.run.smithi077.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-03-12T02:24:46.496 INFO:teuthology.orchestra.run.smithi077.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-03-12T02:24:46.496 INFO:teuthology.orchestra.run.smithi077.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-03-12T02:24:46.497 INFO:teuthology.orchestra.run.smithi077.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-03-12T02:24:46.497 INFO:teuthology.orchestra.run.smithi077.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-03-12T02:24:46.497 INFO:teuthology.orchestra.run.smithi077.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-03-12T02:24:46.497 INFO:teuthology.orchestra.run.smithi077.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-03-12T02:24:46.498 INFO:teuthology.orchestra.run.smithi114.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-03-12T02:24:46.534 INFO:teuthology.orchestra.run.smithi077.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-03-12T02:24:46.534 INFO:teuthology.orchestra.run.smithi077.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-03-12T02:24:46.584 INFO:teuthology.orchestra.run.smithi114.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-03-12T02:24:46.584 INFO:teuthology.orchestra.run.smithi114.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-03-12T02:24:46.805 INFO:teuthology.orchestra.run.smithi077.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-03-12T02:24:46.852 INFO:teuthology.orchestra.run.smithi077.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-03-12T02:24:46.855 INFO:teuthology.orchestra.run.smithi077.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-03-12T02:24:46.867 INFO:teuthology.orchestra.run.smithi114.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-03-12T02:24:46.883 INFO:teuthology.orchestra.run.smithi077.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-03-12T02:24:46.883 INFO:teuthology.orchestra.run.smithi077.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-03-12T02:24:46.883 INFO:teuthology.orchestra.run.smithi077.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.3 [28.8 kB] 2021-03-12T02:24:46.884 INFO:teuthology.orchestra.run.smithi077.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-03-12T02:24:46.884 INFO:teuthology.orchestra.run.smithi077.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-03-12T02:24:46.885 INFO:teuthology.orchestra.run.smithi077.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-03-12T02:24:46.885 INFO:teuthology.orchestra.run.smithi077.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-03-12T02:24:46.900 INFO:teuthology.orchestra.run.smithi114.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-03-12T02:24:46.902 INFO:teuthology.orchestra.run.smithi114.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-03-12T02:24:46.942 INFO:teuthology.orchestra.run.smithi114.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-03-12T02:24:46.943 INFO:teuthology.orchestra.run.smithi114.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-03-12T02:24:46.943 INFO:teuthology.orchestra.run.smithi114.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.3 [28.8 kB] 2021-03-12T02:24:46.944 INFO:teuthology.orchestra.run.smithi114.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-03-12T02:24:46.944 INFO:teuthology.orchestra.run.smithi114.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-03-12T02:24:46.945 INFO:teuthology.orchestra.run.smithi114.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-03-12T02:24:46.945 INFO:teuthology.orchestra.run.smithi114.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-03-12T02:24:46.964 INFO:teuthology.orchestra.run.smithi077.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-03-12T02:24:46.966 INFO:teuthology.orchestra.run.smithi077.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-03-12T02:24:46.973 INFO:teuthology.orchestra.run.smithi077.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-03-12T02:24:46.973 INFO:teuthology.orchestra.run.smithi077.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-03-12T02:24:46.978 INFO:teuthology.orchestra.run.smithi077.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-03-12T02:24:47.024 INFO:teuthology.orchestra.run.smithi114.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-03-12T02:24:47.027 INFO:teuthology.orchestra.run.smithi114.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-03-12T02:24:47.033 INFO:teuthology.orchestra.run.smithi114.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-03-12T02:24:47.037 INFO:teuthology.orchestra.run.smithi114.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-03-12T02:24:47.038 INFO:teuthology.orchestra.run.smithi114.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-03-12T02:24:47.042 INFO:teuthology.orchestra.run.smithi077.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-03-12T02:24:47.043 INFO:teuthology.orchestra.run.smithi077.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-03-12T02:24:47.046 INFO:teuthology.orchestra.run.smithi077.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-03-12T02:24:47.050 INFO:teuthology.orchestra.run.smithi077.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-03-12T02:24:47.051 INFO:teuthology.orchestra.run.smithi077.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-03-12T02:24:47.102 INFO:teuthology.orchestra.run.smithi114.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-03-12T02:24:47.103 INFO:teuthology.orchestra.run.smithi114.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-03-12T02:24:47.109 INFO:teuthology.orchestra.run.smithi114.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-03-12T02:24:47.117 INFO:teuthology.orchestra.run.smithi114.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-03-12T02:24:47.119 INFO:teuthology.orchestra.run.smithi114.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-03-12T02:24:47.120 INFO:teuthology.orchestra.run.smithi077.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-03-12T02:24:47.127 INFO:teuthology.orchestra.run.smithi077.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-03-12T02:24:47.134 INFO:teuthology.orchestra.run.smithi077.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-03-12T02:24:47.137 INFO:teuthology.orchestra.run.smithi077.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-03-12T02:24:47.137 INFO:teuthology.orchestra.run.smithi077.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-03-12T02:24:47.182 INFO:teuthology.orchestra.run.smithi114.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-03-12T02:24:47.188 INFO:teuthology.orchestra.run.smithi114.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-03-12T02:24:47.194 INFO:teuthology.orchestra.run.smithi114.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-03-12T02:24:47.198 INFO:teuthology.orchestra.run.smithi114.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-03-12T02:24:47.201 INFO:teuthology.orchestra.run.smithi114.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-03-12T02:24:47.201 INFO:teuthology.orchestra.run.smithi077.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-03-12T02:24:47.202 INFO:teuthology.orchestra.run.smithi077.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-03-12T02:24:47.202 INFO:teuthology.orchestra.run.smithi077.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-03-12T02:24:47.205 INFO:teuthology.orchestra.run.smithi077.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-03-12T02:24:47.206 INFO:teuthology.orchestra.run.smithi077.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-03-12T02:24:47.261 INFO:teuthology.orchestra.run.smithi114.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-03-12T02:24:47.261 INFO:teuthology.orchestra.run.smithi114.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-03-12T02:24:47.263 INFO:teuthology.orchestra.run.smithi114.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-03-12T02:24:47.266 INFO:teuthology.orchestra.run.smithi114.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-03-12T02:24:47.267 INFO:teuthology.orchestra.run.smithi114.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-03-12T02:24:47.277 INFO:teuthology.orchestra.run.smithi077.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-03-12T02:24:47.277 INFO:teuthology.orchestra.run.smithi077.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-03-12T02:24:47.280 INFO:teuthology.orchestra.run.smithi077.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-03-12T02:24:47.301 INFO:teuthology.orchestra.run.smithi077.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-03-12T02:24:47.301 INFO:teuthology.orchestra.run.smithi077.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-03-12T02:24:47.340 INFO:teuthology.orchestra.run.smithi114.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-03-12T02:24:47.341 INFO:teuthology.orchestra.run.smithi114.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-03-12T02:24:47.344 INFO:teuthology.orchestra.run.smithi114.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-03-12T02:24:47.356 INFO:teuthology.orchestra.run.smithi077.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-03-12T02:24:47.364 INFO:teuthology.orchestra.run.smithi114.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-03-12T02:24:47.365 INFO:teuthology.orchestra.run.smithi114.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-03-12T02:24:47.420 INFO:teuthology.orchestra.run.smithi114.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-03-12T02:24:47.421 INFO:teuthology.orchestra.run.smithi077.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.2.9-120-g3cd904d7-1bionic [5,175 kB] 2021-03-12T02:24:47.453 INFO:teuthology.orchestra.run.smithi077.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-03-12T02:24:47.453 INFO:teuthology.orchestra.run.smithi077.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-03-12T02:24:47.454 INFO:teuthology.orchestra.run.smithi077.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.5 [330 kB] 2021-03-12T02:24:47.458 INFO:teuthology.orchestra.run.smithi077.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-03-12T02:24:47.458 INFO:teuthology.orchestra.run.smithi077.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-03-12T02:24:47.459 INFO:teuthology.orchestra.run.smithi077.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-03-12T02:24:47.459 INFO:teuthology.orchestra.run.smithi077.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-03-12T02:24:47.460 INFO:teuthology.orchestra.run.smithi077.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-03-12T02:24:47.461 INFO:teuthology.orchestra.run.smithi114.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.2.9-120-g3cd904d7-1bionic [5,175 kB] 2021-03-12T02:24:47.461 INFO:teuthology.orchestra.run.smithi077.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-03-12T02:24:47.523 INFO:teuthology.orchestra.run.smithi114.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-03-12T02:24:47.524 INFO:teuthology.orchestra.run.smithi114.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-03-12T02:24:47.524 INFO:teuthology.orchestra.run.smithi114.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.5 [330 kB] 2021-03-12T02:24:47.529 INFO:teuthology.orchestra.run.smithi114.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-03-12T02:24:47.530 INFO:teuthology.orchestra.run.smithi114.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-03-12T02:24:47.530 INFO:teuthology.orchestra.run.smithi114.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-03-12T02:24:47.530 INFO:teuthology.orchestra.run.smithi114.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-03-12T02:24:47.532 INFO:teuthology.orchestra.run.smithi114.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-03-12T02:24:47.562 INFO:teuthology.orchestra.run.smithi114.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-03-12T02:24:47.759 INFO:teuthology.orchestra.run.smithi077.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 15.2.9-120-g3cd904d7-1bionic [161 kB] 2021-03-12T02:24:47.764 INFO:teuthology.orchestra.run.smithi077.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.2.9-120-g3cd904d7-1bionic [1,285 kB] 2021-03-12T02:24:47.801 INFO:teuthology.orchestra.run.smithi114.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 15.2.9-120-g3cd904d7-1bionic [161 kB] 2021-03-12T02:24:47.806 INFO:teuthology.orchestra.run.smithi114.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.2.9-120-g3cd904d7-1bionic [1,285 kB] 2021-03-12T02:24:47.841 INFO:teuthology.orchestra.run.smithi077.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.2.9-120-g3cd904d7-1bionic [5,943 kB] 2021-03-12T02:24:47.918 INFO:teuthology.orchestra.run.smithi114.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.2.9-120-g3cd904d7-1bionic [5,943 kB] 2021-03-12T02:24:48.248 INFO:teuthology.orchestra.run.smithi077.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.2.9-120-g3cd904d7-1bionic [22.8 MB] 2021-03-12T02:24:48.293 INFO:teuthology.orchestra.run.smithi114.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.2.9-120-g3cd904d7-1bionic [22.8 MB] 2021-03-12T02:24:49.767 INFO:teuthology.orchestra.run.smithi077.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.2.9-120-g3cd904d7-1bionic [3,936 B] 2021-03-12T02:24:49.767 INFO:teuthology.orchestra.run.smithi077.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.2.9-120-g3cd904d7-1bionic [516 kB] 2021-03-12T02:24:49.804 INFO:teuthology.orchestra.run.smithi114.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.2.9-120-g3cd904d7-1bionic [3,936 B] 2021-03-12T02:24:49.804 INFO:teuthology.orchestra.run.smithi114.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.2.9-120-g3cd904d7-1bionic [516 kB] 2021-03-12T02:24:49.809 INFO:teuthology.orchestra.run.smithi114.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.2.9-120-g3cd904d7-1bionic [1,853 kB] 2021-03-12T02:24:49.820 INFO:teuthology.orchestra.run.smithi077.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.2.9-120-g3cd904d7-1bionic [1,853 kB] 2021-03-12T02:24:49.949 INFO:teuthology.orchestra.run.smithi077.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 15.2.9-120-g3cd904d7-1bionic [52.2 kB] 2021-03-12T02:24:49.950 INFO:teuthology.orchestra.run.smithi077.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 15.2.9-120-g3cd904d7-1bionic [57.9 kB] 2021-03-12T02:24:49.952 INFO:teuthology.orchestra.run.smithi077.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.2.9-120-g3cd904d7-1bionic [3,197 kB] 2021-03-12T02:24:50.001 INFO:teuthology.orchestra.run.smithi114.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 15.2.9-120-g3cd904d7-1bionic [52.2 kB] 2021-03-12T02:24:50.002 INFO:teuthology.orchestra.run.smithi114.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 15.2.9-120-g3cd904d7-1bionic [57.9 kB] 2021-03-12T02:24:50.002 INFO:teuthology.orchestra.run.smithi114.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.2.9-120-g3cd904d7-1bionic [3,197 kB] 2021-03-12T02:24:50.159 INFO:teuthology.orchestra.run.smithi114.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.2.9-120-g3cd904d7-1bionic [34.3 kB] 2021-03-12T02:24:50.160 INFO:teuthology.orchestra.run.smithi114.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.2.9-120-g3cd904d7-1bionic [7,875 kB] 2021-03-12T02:24:50.218 INFO:teuthology.orchestra.run.smithi077.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.2.9-120-g3cd904d7-1bionic [34.3 kB] 2021-03-12T02:24:50.219 INFO:teuthology.orchestra.run.smithi077.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.2.9-120-g3cd904d7-1bionic [7,875 kB] 2021-03-12T02:24:50.638 INFO:teuthology.orchestra.run.smithi114.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.2.9-120-g3cd904d7-1bionic [4,052 B] 2021-03-12T02:24:50.639 INFO:teuthology.orchestra.run.smithi114.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.2.9-120-g3cd904d7-1bionic [22.5 kB] 2021-03-12T02:24:50.640 INFO:teuthology.orchestra.run.smithi114.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.2.9-120-g3cd904d7-1bionic [44.1 MB] 2021-03-12T02:24:50.890 INFO:teuthology.orchestra.run.smithi077.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.2.9-120-g3cd904d7-1bionic [4,052 B] 2021-03-12T02:24:50.890 INFO:teuthology.orchestra.run.smithi077.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.2.9-120-g3cd904d7-1bionic [22.5 kB] 2021-03-12T02:24:50.891 INFO:teuthology.orchestra.run.smithi077.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.2.9-120-g3cd904d7-1bionic [44.1 MB] 2021-03-12T02:24:53.467 INFO:teuthology.orchestra.run.smithi114.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.2.9-120-g3cd904d7-1bionic [18.0 kB] 2021-03-12T02:24:53.468 INFO:teuthology.orchestra.run.smithi114.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.2.9-120-g3cd904d7-1bionic [8,883 kB] 2021-03-12T02:24:54.044 INFO:teuthology.orchestra.run.smithi114.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.2.9-120-g3cd904d7-1bionic [63.0 kB] 2021-03-12T02:24:54.331 INFO:teuthology.orchestra.run.smithi077.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.2.9-120-g3cd904d7-1bionic [18.0 kB] 2021-03-12T02:24:54.331 INFO:teuthology.orchestra.run.smithi077.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.2.9-120-g3cd904d7-1bionic [8,883 kB] 2021-03-12T02:24:54.713 INFO:teuthology.orchestra.run.smithi077.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-yuri4-testing-2021-03-09-1006-octopus/3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.2.9-120-g3cd904d7-1bionic [63.0 kB] 2021-03-12T02:24:54.762 INFO:teuthology.orchestra.run.smithi114.stdout:Fetched 156 MB in 9s (18.1 MB/s) 2021-03-12T02:24:54.906 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-uritemplate. 2021-03-12T02:24:54.943 INFO:teuthology.orchestra.run.smithi114.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102065 files and directories currently installed.) 2021-03-12T02:24:54.946 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-03-12T02:24:54.988 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-03-12T02:24:55.316 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-03-12T02:24:55.328 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-03-12T02:24:55.408 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-12T02:24:55.426 INFO:teuthology.orchestra.run.smithi077.stdout:Fetched 156 MB in 9s (16.8 MB/s) 2021-03-12T02:24:55.577 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-uritemplate. 2021-03-12T02:24:55.610 INFO:teuthology.orchestra.run.smithi077.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102065 files and directories currently installed.) 2021-03-12T02:24:55.613 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-03-12T02:24:55.654 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-03-12T02:24:55.748 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../002-librbd1_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:24:55.853 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking librbd1 (15.2.9-120-g3cd904d7-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-12T02:24:55.994 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-03-12T02:24:56.007 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-03-12T02:24:56.132 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-12T02:24:56.411 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../003-librados2_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:24:56.471 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../002-librbd1_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:24:56.532 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking librados2 (15.2.9-120-g3cd904d7-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-12T02:24:56.594 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking librbd1 (15.2.9-120-g3cd904d7-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-12T02:24:57.112 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../003-librados2_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:24:57.239 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking librados2 (15.2.9-120-g3cd904d7-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-12T02:24:57.294 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libcephfs2. 2021-03-12T02:24:57.305 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../004-libcephfs2_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:24:57.344 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libcephfs2 (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:24:57.621 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-rados. 2021-03-12T02:24:57.632 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../005-python3-rados_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:24:57.671 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-rados (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:24:57.834 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libcephfs2. 2021-03-12T02:24:57.846 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../004-libcephfs2_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:24:57.876 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libcephfs2 (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:24:58.016 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-03-12T02:24:58.028 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../006-python3-ceph-argparse_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:24:58.066 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-ceph-argparse (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:24:58.145 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-rados. 2021-03-12T02:24:58.156 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../005-python3-rados_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:24:58.195 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-rados (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:24:58.376 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-cephfs. 2021-03-12T02:24:58.391 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../007-python3-cephfs_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:24:58.426 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-cephfs (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:24:58.522 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-03-12T02:24:58.535 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../006-python3-ceph-argparse_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:24:58.574 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-ceph-argparse (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:24:58.754 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-ceph-common. 2021-03-12T02:24:58.767 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../008-python3-ceph-common_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:24:58.804 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-ceph-common (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:24:58.908 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-cephfs. 2021-03-12T02:24:58.919 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../007-python3-cephfs_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:24:58.958 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-cephfs (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:24:59.099 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-prettytable. 2021-03-12T02:24:59.111 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2021-03-12T02:24:59.149 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-03-12T02:24:59.244 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-ceph-common. 2021-03-12T02:24:59.254 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../008-python3-ceph-common_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:24:59.286 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-ceph-common (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:24:59.393 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-rbd. 2021-03-12T02:24:59.405 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../010-python3-rbd_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:24:59.443 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-rbd (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:24:59.564 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-prettytable. 2021-03-12T02:24:59.575 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2021-03-12T02:24:59.614 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-03-12T02:24:59.812 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-03-12T02:24:59.823 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-03-12T02:24:59.849 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-rbd. 2021-03-12T02:24:59.860 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../010-python3-rbd_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:24:59.862 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-12T02:24:59.899 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-rbd (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:00.190 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-03-12T02:25:00.202 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-03-12T02:25:00.231 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-12T02:25:00.268 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-03-12T02:25:00.281 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-03-12T02:25:00.318 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-12T02:25:00.567 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package librgw2. 2021-03-12T02:25:00.579 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../013-librgw2_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:00.617 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking librgw2 (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:00.629 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-03-12T02:25:00.641 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-03-12T02:25:00.679 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-12T02:25:01.007 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package librgw2. 2021-03-12T02:25:01.018 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../013-librgw2_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:01.057 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking librgw2 (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:01.144 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-rgw. 2021-03-12T02:25:01.155 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../014-python3-rgw_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:01.195 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-rgw (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:01.522 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-03-12T02:25:01.534 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-03-12T02:25:01.578 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-rgw. 2021-03-12T02:25:01.588 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../014-python3-rgw_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:01.589 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-12T02:25:01.626 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-rgw (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:01.945 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-03-12T02:25:01.956 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-03-12T02:25:01.996 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-12T02:25:02.058 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-03-12T02:25:02.071 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2021-03-12T02:25:02.108 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-03-12T02:25:02.290 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-03-12T02:25:02.300 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2021-03-12T02:25:02.332 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-03-12T02:25:02.478 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package liboath0. 2021-03-12T02:25:02.490 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2021-03-12T02:25:02.528 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-03-12T02:25:02.711 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package liboath0. 2021-03-12T02:25:02.721 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2021-03-12T02:25:02.759 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-03-12T02:25:02.847 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libradosstriper1. 2021-03-12T02:25:02.859 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../018-libradosstriper1_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:02.897 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libradosstriper1 (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:03.062 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libradosstriper1. 2021-03-12T02:25:03.074 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../018-libradosstriper1_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:03.111 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libradosstriper1 (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:03.284 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-common. 2021-03-12T02:25:03.296 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../019-ceph-common_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:03.333 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-common (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:03.581 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-common. 2021-03-12T02:25:03.594 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../019-ceph-common_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:03.631 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-common (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:05.089 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-base. 2021-03-12T02:25:05.100 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../020-ceph-base_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:05.148 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-base (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:05.394 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-base. 2021-03-12T02:25:05.406 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../020-ceph-base_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:05.453 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-base (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:05.993 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-dateutil. 2021-03-12T02:25:06.006 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2021-03-12T02:25:06.043 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-03-12T02:25:06.316 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-dateutil. 2021-03-12T02:25:06.327 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2021-03-12T02:25:06.337 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-asn1crypto. 2021-03-12T02:25:06.350 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2021-03-12T02:25:06.366 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-03-12T02:25:06.396 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-03-12T02:25:06.640 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-cffi-backend. 2021-03-12T02:25:06.652 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-03-12T02:25:06.690 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-03-12T02:25:06.693 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-asn1crypto. 2021-03-12T02:25:06.706 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2021-03-12T02:25:06.744 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-03-12T02:25:07.004 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-cffi-backend. 2021-03-12T02:25:07.015 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-03-12T02:25:07.026 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-cryptography. 2021-03-12T02:25:07.038 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-03-12T02:25:07.054 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-03-12T02:25:07.076 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-12T02:25:07.340 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-cryptography. 2021-03-12T02:25:07.351 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-03-12T02:25:07.390 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-12T02:25:07.446 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-openssl. 2021-03-12T02:25:07.457 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-03-12T02:25:07.498 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-03-12T02:25:07.743 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-openssl. 2021-03-12T02:25:07.748 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-03-12T02:25:07.757 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-03-12T02:25:07.759 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:25:07.793 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-03-12T02:25:07.798 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mgr-modules-core (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:08.087 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-03-12T02:25:08.099 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:25:08.137 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr-modules-core (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:08.144 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-bcrypt. 2021-03-12T02:25:08.156 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-03-12T02:25:08.194 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-03-12T02:25:08.483 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-bcrypt. 2021-03-12T02:25:08.496 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-03-12T02:25:08.505 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-cherrypy3. 2021-03-12T02:25:08.516 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2021-03-12T02:25:08.533 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-03-12T02:25:08.555 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-03-12T02:25:08.819 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-cherrypy3. 2021-03-12T02:25:08.832 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2021-03-12T02:25:08.868 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-03-12T02:25:08.884 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-jwt. 2021-03-12T02:25:08.895 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-03-12T02:25:08.935 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-03-12T02:25:09.213 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-jwt. 2021-03-12T02:25:09.225 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-03-12T02:25:09.244 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-logutils. 2021-03-12T02:25:09.256 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2021-03-12T02:25:09.263 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-03-12T02:25:09.294 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-03-12T02:25:09.580 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-markupsafe. 2021-03-12T02:25:09.583 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-logutils. 2021-03-12T02:25:09.593 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-03-12T02:25:09.596 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2021-03-12T02:25:09.622 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-03-12T02:25:09.633 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-03-12T02:25:09.924 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-mako. 2021-03-12T02:25:09.935 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-markupsafe. 2021-03-12T02:25:09.936 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2021-03-12T02:25:09.947 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-03-12T02:25:09.975 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-03-12T02:25:09.985 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-03-12T02:25:10.265 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-mako. 2021-03-12T02:25:10.274 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2021-03-12T02:25:10.286 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-simplegeneric. 2021-03-12T02:25:10.297 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2021-03-12T02:25:10.329 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-03-12T02:25:10.345 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-03-12T02:25:10.616 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-simplegeneric. 2021-03-12T02:25:10.627 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2021-03-12T02:25:10.646 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-singledispatch. 2021-03-12T02:25:10.658 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-03-12T02:25:10.665 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-03-12T02:25:10.696 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-03-12T02:25:10.950 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-singledispatch. 2021-03-12T02:25:10.964 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-03-12T02:25:10.990 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-webob. 2021-03-12T02:25:11.000 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-03-12T02:25:11.003 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-03-12T02:25:11.040 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-03-12T02:25:11.312 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-webob. 2021-03-12T02:25:11.323 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-03-12T02:25:11.360 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-bs4. 2021-03-12T02:25:11.361 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-03-12T02:25:11.372 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2021-03-12T02:25:11.401 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-03-12T02:25:11.672 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-bs4. 2021-03-12T02:25:11.684 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2021-03-12T02:25:11.697 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-waitress. 2021-03-12T02:25:11.708 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2021-03-12T02:25:11.723 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-03-12T02:25:11.746 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-03-12T02:25:12.042 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-waitress. 2021-03-12T02:25:12.048 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-tempita. 2021-03-12T02:25:12.055 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2021-03-12T02:25:12.062 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2021-03-12T02:25:12.092 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-03-12T02:25:12.098 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-03-12T02:25:12.403 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-tempita. 2021-03-12T02:25:12.415 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2021-03-12T02:25:12.419 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-paste. 2021-03-12T02:25:12.429 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-03-12T02:25:12.452 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-03-12T02:25:12.468 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-12T02:25:12.745 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-paste. 2021-03-12T02:25:12.758 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-03-12T02:25:12.795 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-12T02:25:12.796 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-03-12T02:25:12.809 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-03-12T02:25:12.846 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-03-12T02:25:13.115 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-03-12T02:25:13.128 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-03-12T02:25:13.142 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pastedeploy. 2021-03-12T02:25:13.153 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2021-03-12T02:25:13.164 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-03-12T02:25:13.274 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-03-12T02:25:13.451 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pastedeploy. 2021-03-12T02:25:13.464 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2021-03-12T02:25:13.501 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-03-12T02:25:13.577 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-webtest. 2021-03-12T02:25:13.588 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-03-12T02:25:13.627 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-03-12T02:25:13.753 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-webtest. 2021-03-12T02:25:13.766 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-03-12T02:25:13.803 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-03-12T02:25:13.980 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pecan. 2021-03-12T02:25:13.992 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2021-03-12T02:25:14.030 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-03-12T02:25:14.131 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pecan. 2021-03-12T02:25:14.143 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2021-03-12T02:25:14.181 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-03-12T02:25:14.395 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-werkzeug. 2021-03-12T02:25:14.404 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-03-12T02:25:14.443 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-12T02:25:14.535 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-werkzeug. 2021-03-12T02:25:14.548 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-03-12T02:25:14.585 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-12T02:25:14.804 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mgr. 2021-03-12T02:25:14.816 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../045-ceph-mgr_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:14.854 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mgr (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:14.955 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr. 2021-03-12T02:25:14.968 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../045-ceph-mgr_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:15.014 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:15.314 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mon. 2021-03-12T02:25:15.325 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../046-ceph-mon_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:15.365 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mon (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:15.476 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mon. 2021-03-12T02:25:15.487 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../046-ceph-mon_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:15.592 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mon (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:16.175 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-osd. 2021-03-12T02:25:16.188 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../047-ceph-osd_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:16.217 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-osd (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:16.896 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-osd. 2021-03-12T02:25:16.910 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../047-ceph-osd_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:17.204 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-osd (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:18.622 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph. 2021-03-12T02:25:19.733 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph. 2021-03-12T02:25:19.733 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../048-ceph_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:19.734 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:19.734 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-fuse. 2021-03-12T02:25:19.734 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../049-ceph-fuse_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:19.734 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-fuse (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:19.735 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../048-ceph_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:19.735 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:19.735 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-fuse. 2021-03-12T02:25:19.736 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../049-ceph-fuse_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:19.736 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-fuse (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:19.736 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mds. 2021-03-12T02:25:19.736 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../050-ceph-mds_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:19.737 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mds (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:19.899 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mds. 2021-03-12T02:25:19.901 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package cephadm. 2021-03-12T02:25:19.911 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../050-ceph-mds_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:19.913 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../051-cephadm_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:19.949 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mds (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:19.959 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking cephadm (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:20.279 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-jinja2. 2021-03-12T02:25:20.290 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-03-12T02:25:20.328 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-12T02:25:20.435 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package cephadm. 2021-03-12T02:25:20.447 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../051-cephadm_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:20.485 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking cephadm (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:20.640 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-03-12T02:25:20.653 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:25:20.689 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mgr-cephadm (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:20.771 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-jinja2. 2021-03-12T02:25:20.784 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-03-12T02:25:20.821 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-12T02:25:21.009 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-repoze.lru. 2021-03-12T02:25:21.022 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2021-03-12T02:25:21.059 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-03-12T02:25:21.140 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-03-12T02:25:21.153 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:25:21.192 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr-cephadm (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:21.337 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-routes. 2021-03-12T02:25:21.349 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2021-03-12T02:25:21.378 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-03-12T02:25:21.527 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-repoze.lru. 2021-03-12T02:25:21.539 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2021-03-12T02:25:21.681 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-03-12T02:25:21.693 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:25:21.731 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mgr-dashboard (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:21.738 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-03-12T02:25:22.080 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-routes. 2021-03-12T02:25:22.091 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2021-03-12T02:25:22.130 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-03-12T02:25:22.449 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-03-12T02:25:22.461 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:25:22.499 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr-dashboard (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:22.626 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2021-03-12T02:25:22.639 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../057-ceph-mgr-diskprediction-cloud_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:25:22.676 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:22.979 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-numpy. 2021-03-12T02:25:22.991 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-03-12T02:25:23.029 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-12T02:25:23.411 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2021-03-12T02:25:23.424 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../057-ceph-mgr-diskprediction-cloud_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:25:23.461 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:23.618 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-decorator. 2021-03-12T02:25:23.631 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-03-12T02:25:23.670 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-03-12T02:25:23.772 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-numpy. 2021-03-12T02:25:23.785 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-03-12T02:25:23.822 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-12T02:25:23.982 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-scipy. 2021-03-12T02:25:23.998 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-03-12T02:25:24.040 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-03-12T02:25:24.426 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-decorator. 2021-03-12T02:25:24.437 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-03-12T02:25:24.468 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-03-12T02:25:24.762 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-scipy. 2021-03-12T02:25:24.774 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-03-12T02:25:24.812 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-03-12T02:25:25.173 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-03-12T02:25:25.191 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-03-12T02:25:25.232 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-03-12T02:25:25.676 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-joblib. 2021-03-12T02:25:25.689 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-03-12T02:25:25.726 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-joblib (0.11-1) ... 2021-03-12T02:25:25.938 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-03-12T02:25:25.951 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-03-12T02:25:25.996 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-03-12T02:25:26.047 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-sklearn. 2021-03-12T02:25:26.052 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-03-12T02:25:26.087 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-03-12T02:25:26.475 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-joblib. 2021-03-12T02:25:26.487 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-03-12T02:25:26.531 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-joblib (0.11-1) ... 2021-03-12T02:25:26.644 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-03-12T02:25:26.657 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:25:26.694 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:26.911 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-sklearn. 2021-03-12T02:25:26.923 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-03-12T02:25:26.961 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-03-12T02:25:27.485 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-03-12T02:25:27.498 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:25:27.535 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:27.830 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-monotonic. 2021-03-12T02:25:27.844 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-03-12T02:25:27.880 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-03-12T02:25:28.192 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-fasteners. 2021-03-12T02:25:28.204 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-03-12T02:25:28.242 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-03-12T02:25:28.552 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-httplib2. 2021-03-12T02:25:28.566 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb ... 2021-03-12T02:25:28.588 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-monotonic. 2021-03-12T02:25:28.599 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-03-12T02:25:28.602 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-12T02:25:28.632 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-03-12T02:25:28.905 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pyasn1. 2021-03-12T02:25:28.917 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-03-12T02:25:28.955 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-03-12T02:25:28.957 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-fasteners. 2021-03-12T02:25:28.970 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-03-12T02:25:29.008 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-03-12T02:25:29.294 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-httplib2. 2021-03-12T02:25:29.303 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-03-12T02:25:29.308 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb ... 2021-03-12T02:25:29.313 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-03-12T02:25:29.350 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-12T02:25:29.353 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-12T02:25:29.615 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pyasn1. 2021-03-12T02:25:29.626 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-03-12T02:25:29.645 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-rsa. 2021-03-12T02:25:29.657 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-03-12T02:25:29.663 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-03-12T02:25:29.695 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-03-12T02:25:29.967 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-03-12T02:25:29.979 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-03-12T02:25:29.998 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-03-12T02:25:30.009 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-03-12T02:25:30.025 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-12T02:25:30.048 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-12T02:25:30.337 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-rsa. 2021-03-12T02:25:30.349 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-03-12T02:25:30.386 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-03-12T02:25:30.486 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-oauth2client. 2021-03-12T02:25:30.503 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-03-12T02:25:30.535 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-03-12T02:25:30.699 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-03-12T02:25:30.711 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-03-12T02:25:30.748 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-12T02:25:30.871 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-setuptools. 2021-03-12T02:25:30.884 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-03-12T02:25:30.921 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-03-12T02:25:31.194 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-oauth2client. 2021-03-12T02:25:31.209 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-03-12T02:25:31.245 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-03-12T02:25:31.325 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-websocket. 2021-03-12T02:25:31.337 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-03-12T02:25:31.375 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-03-12T02:25:31.522 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-setuptools. 2021-03-12T02:25:31.536 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-03-12T02:25:31.581 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-03-12T02:25:31.690 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-kubernetes. 2021-03-12T02:25:31.699 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-03-12T02:25:31.736 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-12T02:25:32.026 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-websocket. 2021-03-12T02:25:32.038 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-03-12T02:25:32.068 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-03-12T02:25:32.124 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-03-12T02:25:32.136 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:25:32.173 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mgr-k8sevents (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:32.346 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-kubernetes. 2021-03-12T02:25:32.359 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-03-12T02:25:32.404 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-12T02:25:32.493 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-json-pointer. 2021-03-12T02:25:32.518 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-03-12T02:25:32.792 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-03-12T02:25:32.825 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-03-12T02:25:32.843 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:25:32.875 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr-k8sevents (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:33.145 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-jsonpatch. 2021-03-12T02:25:33.157 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-03-12T02:25:33.194 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-json-pointer. 2021-03-12T02:25:33.198 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-12T02:25:33.207 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-03-12T02:25:33.244 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-03-12T02:25:33.448 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-03-12T02:25:33.460 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../079-ceph-mgr-rook_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:25:33.498 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-mgr-rook (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:33.606 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-jsonpatch. 2021-03-12T02:25:33.618 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-03-12T02:25:33.658 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-12T02:25:33.860 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libonig4:amd64. 2021-03-12T02:25:33.872 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-03-12T02:25:33.934 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-03-12T02:25:33.948 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../079-ceph-mgr-rook_15.2.9-120-g3cd904d7-1bionic_all.deb ... 2021-03-12T02:25:33.986 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-03-12T02:25:33.992 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr-rook (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:34.303 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libonig4:amd64. 2021-03-12T02:25:34.315 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-03-12T02:25:34.349 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libjq1:amd64. 2021-03-12T02:25:34.360 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-03-12T02:25:34.397 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-03-12T02:25:34.438 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-03-12T02:25:34.625 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package jq. 2021-03-12T02:25:34.639 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-03-12T02:25:34.676 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-03-12T02:25:34.791 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libjq1:amd64. 2021-03-12T02:25:34.803 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-03-12T02:25:34.841 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-03-12T02:25:34.927 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package socat. 2021-03-12T02:25:34.941 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-03-12T02:25:34.986 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-03-12T02:25:35.094 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package jq. 2021-03-12T02:25:35.107 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-03-12T02:25:35.136 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-03-12T02:25:35.356 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-03-12T02:25:35.368 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-03-12T02:25:35.371 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package socat. 2021-03-12T02:25:35.385 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-03-12T02:25:35.405 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-12T02:25:35.422 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-03-12T02:25:35.692 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package xmlstarlet. 2021-03-12T02:25:35.703 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-03-12T02:25:35.750 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-03-12T02:25:35.842 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-03-12T02:25:35.855 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-03-12T02:25:35.892 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-12T02:25:36.088 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package ceph-test. 2021-03-12T02:25:36.101 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../086-ceph-test_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:36.137 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking ceph-test (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:36.162 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package xmlstarlet. 2021-03-12T02:25:36.174 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-03-12T02:25:36.203 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-03-12T02:25:36.527 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-test. 2021-03-12T02:25:36.538 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../086-ceph-test_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:36.583 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-test (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:39.349 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package libcephfs-dev. 2021-03-12T02:25:39.362 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../087-libcephfs-dev_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:39.399 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking libcephfs-dev (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:39.693 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package nvme-cli. 2021-03-12T02:25:39.708 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-03-12T02:25:39.743 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-03-12T02:25:39.911 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libcephfs-dev. 2021-03-12T02:25:39.926 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../087-libcephfs-dev_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:39.961 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libcephfs-dev (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:40.072 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python-attr. 2021-03-12T02:25:40.086 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-03-12T02:25:40.123 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python-attr (17.4.0-2) ... 2021-03-12T02:25:40.247 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package nvme-cli. 2021-03-12T02:25:40.259 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-03-12T02:25:40.297 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-03-12T02:25:40.409 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python-funcsigs. 2021-03-12T02:25:40.425 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-03-12T02:25:40.458 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-03-12T02:25:40.661 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python-attr. 2021-03-12T02:25:40.672 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-03-12T02:25:40.710 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python-attr (17.4.0-2) ... 2021-03-12T02:25:40.744 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python-pluggy. 2021-03-12T02:25:40.756 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-03-12T02:25:40.794 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-03-12T02:25:41.027 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python-funcsigs. 2021-03-12T02:25:41.034 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-03-12T02:25:41.071 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-03-12T02:25:41.105 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python-py. 2021-03-12T02:25:41.118 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-03-12T02:25:41.155 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python-py (1.5.2-1) ... 2021-03-12T02:25:41.357 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python-pluggy. 2021-03-12T02:25:41.369 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-03-12T02:25:41.407 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-03-12T02:25:41.475 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python-pytest. 2021-03-12T02:25:41.489 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-03-12T02:25:41.533 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-03-12T02:25:41.712 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python-py. 2021-03-12T02:25:41.722 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-03-12T02:25:41.760 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python-py (1.5.2-1) ... 2021-03-12T02:25:41.904 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-attr. 2021-03-12T02:25:41.916 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-03-12T02:25:41.953 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-03-12T02:25:42.080 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python-pytest. 2021-03-12T02:25:42.094 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-03-12T02:25:42.130 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-03-12T02:25:42.256 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-cycler. 2021-03-12T02:25:42.270 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-03-12T02:25:42.314 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-03-12T02:25:42.450 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-attr. 2021-03-12T02:25:42.463 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-03-12T02:25:42.502 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-03-12T02:25:42.626 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-webencodings. 2021-03-12T02:25:42.637 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-03-12T02:25:42.667 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-03-12T02:25:42.805 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-cycler. 2021-03-12T02:25:42.818 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-03-12T02:25:42.936 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-03-12T02:25:42.962 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-html5lib. 2021-03-12T02:25:42.981 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-03-12T02:25:43.015 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-03-12T02:25:43.159 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-webencodings. 2021-03-12T02:25:43.172 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-03-12T02:25:43.211 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-03-12T02:25:43.352 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-03-12T02:25:43.367 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-03-12T02:25:43.401 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-12T02:25:43.523 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-html5lib. 2021-03-12T02:25:43.535 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-03-12T02:25:43.583 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-03-12T02:25:43.847 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pyparsing. 2021-03-12T02:25:43.859 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-03-12T02:25:43.892 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-12T02:25:43.931 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-03-12T02:25:43.950 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-03-12T02:25:43.981 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-12T02:25:44.195 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-tz. 2021-03-12T02:25:44.209 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-03-12T02:25:44.253 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-tz (2018.3-2) ... 2021-03-12T02:25:44.435 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pyparsing. 2021-03-12T02:25:44.447 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-03-12T02:25:44.485 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-12T02:25:44.573 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-matplotlib. 2021-03-12T02:25:44.585 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-03-12T02:25:44.623 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-12T02:25:44.788 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-tz. 2021-03-12T02:25:44.801 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-03-12T02:25:44.837 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-tz (2018.3-2) ... 2021-03-12T02:25:45.141 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-matplotlib. 2021-03-12T02:25:45.155 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-03-12T02:25:45.198 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-12T02:25:45.270 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-olefile. 2021-03-12T02:25:45.284 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-03-12T02:25:45.320 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-03-12T02:25:45.631 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pastescript. 2021-03-12T02:25:45.643 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-03-12T02:25:45.682 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-03-12T02:25:45.829 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-olefile. 2021-03-12T02:25:45.844 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-03-12T02:25:45.887 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-03-12T02:25:46.003 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pil:amd64. 2021-03-12T02:25:46.014 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.5_amd64.deb ... 2021-03-12T02:25:46.053 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.5) ... 2021-03-12T02:25:46.157 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pastescript. 2021-03-12T02:25:46.170 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-03-12T02:25:46.199 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-03-12T02:25:46.435 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pluggy. 2021-03-12T02:25:46.444 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-03-12T02:25:46.481 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-03-12T02:25:46.519 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pil:amd64. 2021-03-12T02:25:46.531 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.5_amd64.deb ... 2021-03-12T02:25:46.569 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.5) ... 2021-03-12T02:25:46.767 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-py. 2021-03-12T02:25:46.781 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-03-12T02:25:46.825 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-py (1.5.2-1) ... 2021-03-12T02:25:46.939 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pluggy. 2021-03-12T02:25:46.951 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-03-12T02:25:46.989 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-03-12T02:25:47.153 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pyinotify. 2021-03-12T02:25:47.167 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-03-12T02:25:47.203 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-03-12T02:25:47.333 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-py. 2021-03-12T02:25:47.346 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-03-12T02:25:47.383 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-py (1.5.2-1) ... 2021-03-12T02:25:47.498 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-pytest. 2021-03-12T02:25:47.515 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-03-12T02:25:47.556 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-03-12T02:25:47.737 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pyinotify. 2021-03-12T02:25:47.750 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-03-12T02:25:47.787 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-03-12T02:25:47.843 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-simplejson. 2021-03-12T02:25:47.857 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-03-12T02:25:47.920 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-03-12T02:25:48.090 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pytest. 2021-03-12T02:25:48.107 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-03-12T02:25:48.140 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-03-12T02:25:48.238 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package python3-tk:amd64. 2021-03-12T02:25:48.249 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-03-12T02:25:48.287 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-12T02:25:48.451 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-simplejson. 2021-03-12T02:25:48.465 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-03-12T02:25:48.509 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-03-12T02:25:48.741 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package radosgw. 2021-03-12T02:25:48.753 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../111-radosgw_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:48.791 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking radosgw (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:48.813 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-tk:amd64. 2021-03-12T02:25:48.826 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-03-12T02:25:48.854 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-12T02:25:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package radosgw. 2021-03-12T02:25:49.239 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../111-radosgw_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:49.283 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking radosgw (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:49.719 INFO:teuthology.orchestra.run.smithi114.stdout:Selecting previously unselected package rbd-fuse. 2021-03-12T02:25:49.732 INFO:teuthology.orchestra.run.smithi114.stdout:Preparing to unpack .../112-rbd-fuse_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:49.769 INFO:teuthology.orchestra.run.smithi114.stdout:Unpacking rbd-fuse (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:50.096 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-03-12T02:25:50.229 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package rbd-fuse. 2021-03-12T02:25:50.242 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../112-rbd-fuse_15.2.9-120-g3cd904d7-1bionic_amd64.deb ... 2021-03-12T02:25:50.289 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking rbd-fuse (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:50.441 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-12T02:25:50.575 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-03-12T02:25:50.784 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-12T02:25:50.867 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-12T02:25:50.884 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-12T02:25:51.235 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-12T02:25:51.361 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-12T02:25:51.490 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.5) ... 2021-03-12T02:25:51.872 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-03-12T02:25:51.989 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.5) ... 2021-03-12T02:25:52.164 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-webencodings (0.5-2) ... 2021-03-12T02:25:52.449 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python-attr (17.4.0-2) ... 2021-03-12T02:25:52.565 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-03-12T02:25:52.642 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-03-12T02:25:52.744 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-py (1.5.2-1) ... 2021-03-12T02:25:52.833 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-webencodings (0.5-2) ... 2021-03-12T02:25:53.086 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-03-12T02:25:53.117 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python-attr (17.4.0-2) ... 2021-03-12T02:25:53.310 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-03-12T02:25:53.436 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-py (1.5.2-1) ... 2021-03-12T02:25:53.967 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-03-12T02:25:53.968 INFO:teuthology.orchestra.run.smithi114.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-03-12T02:25:53.968 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-03-12T02:25:54.083 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-ceph-argparse (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:54.195 INFO:teuthology.orchestra.run.smithi077.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-03-12T02:25:54.395 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-03-12T02:25:54.499 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-03-12T02:25:54.684 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-03-12T02:25:54.767 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-ceph-argparse (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:54.785 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-12T02:25:55.051 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-03-12T02:25:55.344 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-03-12T02:25:55.478 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-12T02:25:55.721 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-03-12T02:25:56.047 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-03-12T02:25:56.340 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python-py (1.5.2-1) ... 2021-03-12T02:25:56.406 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-03-12T02:25:56.566 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-03-12T02:25:56.699 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-03-12T02:25:56.901 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-03-12T02:25:56.967 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python-py (1.5.2-1) ... 2021-03-12T02:25:56.978 INFO:teuthology.orchestra.run.smithi114.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-03-12T02:25:57.201 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-03-12T02:25:57.219 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-03-12T02:25:57.345 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-03-12T02:25:57.545 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-03-12T02:25:57.546 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-03-12T02:25:57.634 INFO:teuthology.orchestra.run.smithi077.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-03-12T02:25:57.839 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-12T02:25:57.888 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-03-12T02:25:58.005 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-03-12T02:25:58.157 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-03-12T02:25:58.182 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-03-12T02:25:58.441 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-03-12T02:25:58.458 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-12T02:25:58.775 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-03-12T02:25:58.800 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-03-12T02:25:59.060 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-03-12T02:25:59.118 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-03-12T02:25:59.420 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up cephadm (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:25:59.445 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-03-12T02:25:59.796 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-03-12T02:26:00.089 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up cephadm (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:00.103 INFO:teuthology.orchestra.run.smithi114.stdout:Adding system user cephadm....done 2021-03-12T02:26:00.251 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-03-12T02:26:00.510 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-12T02:26:00.637 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-03-12T02:26:00.714 INFO:teuthology.orchestra.run.smithi077.stdout:Adding system user cephadm....done 2021-03-12T02:26:00.813 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-03-12T02:26:00.887 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-03-12T02:26:01.139 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-03-12T02:26:01.188 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-12T02:26:01.331 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-03-12T02:26:01.440 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-03-12T02:26:01.457 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-03-12T02:26:01.566 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-03-12T02:26:01.783 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-03-12T02:26:01.901 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-ceph-common (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:02.068 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-03-12T02:26:02.185 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-03-12T02:26:02.193 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-monotonic (1.1-2) ... 2021-03-12T02:26:02.495 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-12T02:26:02.520 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-ceph-common (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:02.621 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-12T02:26:02.846 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-monotonic (1.1-2) ... 2021-03-12T02:26:02.947 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-03-12T02:26:03.147 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-12T02:26:03.265 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-12T02:26:03.557 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-12T02:26:03.605 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-03-12T02:26:03.646 INFO:teuthology.orchestra.run.smithi114.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-03-12T02:26:03.679 INFO:teuthology.orchestra.run.smithi114.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-03-12T02:26:03.909 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-03-12T02:26:03.926 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-12T02:26:04.016 INFO:teuthology.orchestra.run.smithi077.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-03-12T02:26:04.048 INFO:teuthology.orchestra.run.smithi077.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-03-12T02:26:04.201 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-12T02:26:04.287 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-03-12T02:26:04.486 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-12T02:26:04.588 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-12T02:26:04.820 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-attr (17.4.0-2) ... 2021-03-12T02:26:04.889 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-12T02:26:05.130 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-03-12T02:26:05.224 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-attr (17.4.0-2) ... 2021-03-12T02:26:05.334 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-12T02:26:05.516 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-03-12T02:26:05.692 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-12T02:26:05.749 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-03-12T02:26:05.839 INFO:teuthology.orchestra.run.smithi114.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-03-12T02:26:06.077 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-03-12T02:26:06.126 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up liboath0 (2.6.1-1) ... 2021-03-12T02:26:06.166 INFO:teuthology.orchestra.run.smithi077.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-03-12T02:26:06.294 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-12T02:26:06.420 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-12T02:26:06.421 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up liboath0 (2.6.1-1) ... 2021-03-12T02:26:06.606 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-12T02:26:06.688 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-03-12T02:26:06.731 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-12T02:26:06.814 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-03-12T02:26:06.966 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-03-12T02:26:07.067 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-03-12T02:26:07.190 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-12T02:26:07.316 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-03-12T02:26:07.427 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-12T02:26:07.553 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-03-12T02:26:07.617 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-tz (2018.3-2) ... 2021-03-12T02:26:07.847 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-tz (2018.3-2) ... 2021-03-12T02:26:07.919 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-03-12T02:26:08.147 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-03-12T02:26:08.187 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-03-12T02:26:08.473 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-03-12T02:26:08.496 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-joblib (0.11-1) ... 2021-03-12T02:26:08.783 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-joblib (0.11-1) ... 2021-03-12T02:26:08.864 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-routes (2.4.1-1) ... 2021-03-12T02:26:09.134 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-routes (2.4.1-1) ... 2021-03-12T02:26:09.165 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-03-12T02:26:09.410 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-03-12T02:26:09.457 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-03-12T02:26:09.694 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-03-12T02:26:09.784 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-12T02:26:10.021 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-12T02:26:10.144 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-03-12T02:26:10.381 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-03-12T02:26:10.412 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up jq (1.5+dfsg-2) ... 2021-03-12T02:26:10.538 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-03-12T02:26:10.699 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up jq (1.5+dfsg-2) ... 2021-03-12T02:26:10.800 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-03-12T02:26:11.709 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-12T02:26:11.947 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-12T02:26:12.010 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-03-12T02:26:12.256 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-03-12T02:26:12.328 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-03-12T02:26:12.454 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-03-12T02:26:12.599 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-03-12T02:26:12.700 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-03-12T02:26:12.831 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python-pytest (3.3.2-2) ... 2021-03-12T02:26:13.085 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python-pytest (3.3.2-2) ... 2021-03-12T02:26:13.090 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-03-12T02:26:13.219 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-03-12T02:26:13.320 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-03-12T02:26:13.342 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up librados2 (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:13.446 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-03-12T02:26:13.460 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-03-12T02:26:13.582 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up librados2 (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:13.706 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-03-12T02:26:13.786 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libcephfs2 (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:13.904 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libcephfs-dev (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:14.024 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libcephfs2 (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:14.029 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-rados (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:14.156 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-03-12T02:26:14.157 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libcephfs-dev (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:14.252 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-rados (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:14.353 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-03-12T02:26:14.465 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-03-12T02:26:14.662 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-03-12T02:26:14.766 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-03-12T02:26:14.954 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-03-12T02:26:15.084 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-12T02:26:15.297 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-12T02:26:15.997 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-03-12T02:26:16.218 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-03-12T02:26:16.851 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-03-12T02:26:17.015 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-03-12T02:26:17.136 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up libradosstriper1 (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:17.245 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-cephfs (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:17.308 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libradosstriper1 (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:17.450 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-cephfs (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:17.529 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up librgw2 (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:17.657 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-fuse (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:17.726 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up librgw2 (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:17.827 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-fuse (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:17.866 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-12T02:26:17.866 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-12T02:26:18.033 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-12T02:26:18.034 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-12T02:26:18.123 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up librbd1 (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:18.249 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-03-12T02:26:18.320 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up librbd1 (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:18.339 INFO:teuthology.orchestra.run.smithi114.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-03-12T02:26:18.380 INFO:teuthology.orchestra.run.smithi114.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-03-12T02:26:18.446 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-03-12T02:26:18.536 INFO:teuthology.orchestra.run.smithi077.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-03-12T02:26:18.577 INFO:teuthology.orchestra.run.smithi077.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-03-12T02:26:18.694 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-12T02:26:18.858 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-12T02:26:19.347 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mgr-modules-core (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:19.473 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-03-12T02:26:19.502 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr-modules-core (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:19.611 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-03-12T02:26:19.799 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-rgw (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:19.925 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up rbd-fuse (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:19.955 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-rgw (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:20.051 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up python3-rbd (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:20.081 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up rbd-fuse (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:20.177 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-common (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:20.206 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-rbd (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:20.332 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-common (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:20.703 INFO:teuthology.orchestra.run.smithi114.stdout:Adding group ceph....done 2021-03-12T02:26:20.784 INFO:teuthology.orchestra.run.smithi077.stdout:Adding group ceph....done 2021-03-12T02:26:21.081 INFO:teuthology.orchestra.run.smithi114.stdout:Adding system user ceph....done 2021-03-12T02:26:21.178 INFO:teuthology.orchestra.run.smithi077.stdout:Adding system user ceph....done 2021-03-12T02:26:21.273 INFO:teuthology.orchestra.run.smithi114.stdout:Setting system user ceph properties....done 2021-03-12T02:26:21.378 INFO:teuthology.orchestra.run.smithi077.stdout:Setting system user ceph properties....done 2021-03-12T02:26:21.426 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-03-12T02:26:21.525 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-03-12T02:26:21.627 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-03-12T02:26:21.763 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-03-12T02:26:21.904 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up radosgw (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:22.059 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up radosgw (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:22.170 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-12T02:26:22.170 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-12T02:26:22.313 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-12T02:26:22.313 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-12T02:26:22.539 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-test (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:22.667 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-base (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:22.711 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-test (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:22.837 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-base (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:22.997 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-03-12T02:26:23.251 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-03-12T02:26:23.308 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mgr (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:23.529 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-12T02:26:23.529 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-12T02:26:23.531 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:23.739 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-12T02:26:23.739 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-12T02:26:23.952 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mgr-diskprediction-local (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:24.049 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr-diskprediction-local (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:24.087 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-osd (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:24.192 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-osd (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:24.523 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-12T02:26:24.523 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-12T02:26:24.626 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-12T02:26:24.987 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-12T02:26:24.988 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr-cephadm (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:24.989 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mgr-cephadm (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:24.989 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:25.046 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:25.083 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mgr-rook (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:25.188 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr-rook (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:25.225 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mds (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:25.331 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mds (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:25.446 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-12T02:26:25.446 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-12T02:26:25.555 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-12T02:26:25.556 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-12T02:26:25.735 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mon (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:25.857 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mon (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:25.952 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-12T02:26:25.952 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-12T02:26:26.072 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-12T02:26:26.073 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-12T02:26:26.211 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mgr-k8sevents (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:26.312 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph-mgr-dashboard (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:26.350 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr-k8sevents (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:26.455 INFO:teuthology.orchestra.run.smithi114.stdout:Setting up ceph (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:26.459 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr-dashboard (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:26.577 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph (15.2.9-120-g3cd904d7-1bionic) ... 2021-03-12T02:26:26.582 INFO:teuthology.orchestra.run.smithi114.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-03-12T02:26:26.704 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-03-12T02:26:26.783 INFO:teuthology.orchestra.run.smithi114.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-03-12T02:26:26.923 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-03-12T02:26:28.969 INFO:teuthology.orchestra.run.smithi114.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-03-12T02:26:28.983 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-03-12T02:26:29.012 INFO:teuthology.orchestra.run.smithi114.stdout:ureadahead will be reprofiled on next reboot 2021-03-12T02:26:29.026 INFO:teuthology.orchestra.run.smithi077.stdout:ureadahead will be reprofiled on next reboot 2021-03-12T02:26:29.070 INFO:teuthology.orchestra.run.smithi114.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-03-12T02:26:29.084 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-03-12T02:26:29.509 INFO:teuthology.orchestra.run.smithi077.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-03-12T02:26:29.512 DEBUG:teuthology.parallel:result is None 2021-03-12T02:26:29.516 INFO:teuthology.orchestra.run.smithi114.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-03-12T02:26:29.518 DEBUG:teuthology.parallel:result is None 2021-03-12T02:26:29.518 DEBUG:teuthology.orchestra.run.smithi077:> sudo lsb_release -is 2021-03-12T02:26:29.572 INFO:teuthology.orchestra.run.smithi077.stdout:Ubuntu 2021-03-12T02:26:29.576 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-12T02:26:29.577 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb 2021-03-12T02:26:29.722 DEBUG:teuthology.orchestra.run.smithi077:> dpkg-query -W -f '${Version}' ceph 2021-03-12T02:26:29.748 INFO:teuthology.orchestra.run.smithi077.stdout:15.2.9-120-g3cd904d7-1bionic 2021-03-12T02:26:29.749 INFO:teuthology.packaging:The installed version of ceph is 15.2.9-120-g3cd904d7-1bionic 2021-03-12T02:26:29.749 INFO:teuthology.task.install:The correct ceph version 15.2.9-120-g3cd904d7-1bionic is installed. 2021-03-12T02:26:29.750 DEBUG:teuthology.orchestra.run.smithi114:> sudo lsb_release -is 2021-03-12T02:26:29.804 INFO:teuthology.orchestra.run.smithi114.stdout:Ubuntu 2021-03-12T02:26:29.809 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-12T02:26:29.810 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb 2021-03-12T02:26:29.945 DEBUG:teuthology.orchestra.run.smithi114:> dpkg-query -W -f '${Version}' ceph 2021-03-12T02:26:29.968 INFO:teuthology.orchestra.run.smithi114.stdout:15.2.9-120-g3cd904d7-1bionic 2021-03-12T02:26:29.969 INFO:teuthology.packaging:The installed version of ceph is 15.2.9-120-g3cd904d7-1bionic 2021-03-12T02:26:29.969 INFO:teuthology.task.install:The correct ceph version 15.2.9-120-g3cd904d7-1bionic is installed. 2021-03-12T02:26:29.970 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-03-12T02:26:29.972 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2021-03-12T02:26:29.972 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-12T02:26:29.987 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2021-03-12T02:26:29.987 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-12T02:26:30.030 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-03-12T02:26:30.030 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2021-03-12T02:26:30.031 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2021-03-12T02:26:30.048 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-12T02:26:30.103 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2021-03-12T02:26:30.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2021-03-12T02:26:30.118 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-12T02:26:30.173 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-03-12T02:26:30.174 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2021-03-12T02:26:30.174 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-12T02:26:30.190 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-12T02:26:30.248 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2021-03-12T02:26:30.248 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-12T02:26:30.264 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-12T02:26:30.321 INFO:teuthology.run_tasks:Running task ceph... 2021-03-12T02:26:30.391 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-03-12T02:26:30.394 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /var/log/ceph 2021-03-12T02:26:30.397 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /var/log/ceph 2021-03-12T02:26:30.412 INFO:tasks.ceph:Disabling ceph logrotate... 2021-03-12T02:26:30.413 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/logrotate.d/ceph 2021-03-12T02:26:30.452 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/logrotate.d/ceph 2021-03-12T02:26:30.469 INFO:tasks.ceph:Creating extra log directories... 2021-03-12T02:26:30.469 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-03-12T02:26:30.508 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-03-12T02:26:30.527 INFO:tasks.ceph:Creating ceph cluster c1... 2021-03-12T02:26:30.527 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 1, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-03-12T02:26:30.528 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2021-03-09_20:35:33-rgw-wip-yuri4-testing-2021-03-09-1006-octopus-distro-basic-smithi/5951134', 'branch': 'wip-yuri4-testing-2021-03-09-1006-octopus', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '5951134', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2021-03-09_20:35:33-rgw-wip-yuri4-testing-2021-03-09-1006-octopus-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2021-03-09-1006-octopus'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 1, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-yuri4-testing-2021-03-09-1006-octopus', 'sha1': '3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-yuri4-testing-2021-03-09-1006-octopus', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb', 'targets': {'smithi077.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgWjnm/iuIIjyr4cN1NZVVqXGxv1uNGNsn9pMzqAsfjgBCcQPK/LuPILtOZjDQo5bI2ch8/+/Y+iRPbmDlNKZegCoe/E9U0ccVreM3hIU1iM+A7NQf59jE++pSt0z/9KX1ZeD5ValQ79L1wDholUHsR5CKNq8t9uTah2Jv9LzrSAFhIHWD7gsiUtQV5Nh+zxiXElh2GoC6maYlFqBiJBM/SGU/hX8DwhFnXS7e4E4NshtegWeijAt93Zsw4Anpu1ChfqYwB4uJW4EsbZmcx2zpgNbisXclfgtf/udwb4qJQRE8XjO4wWaqCGtNh9s4wFZpSaVQGBvDLqvsaF85L4SN', 'smithi114.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXjz7S6TEnBMVHdLOfCMhKgTnCP9W9q8QDBYiH63QhmXddO2kAhw1oD+J8nw92mrC6qbK66SSkhGRPGDRFzB7MiY1tw4fPWBuzrDC4WDnBFHuD2tOLcb9Lwoue6DlV+bzChQpW6RgRYscpZJpC99lSnLHxX3cI4vS3ScpDBrnGG4NfqFzdKE1Hs8TxOmV4PhqTZTEo08/V7/WKGS8zLfmKPm1/bPgbbl/cJ+xFfYc0ro8wd6aQmdK+eZJx7DYK13Lt27YHVLq/aeRBkuggXLYZHz4gQAwXwVf5NWq9sixagCZe5TzjdAdQjgPWXl63/W2SUSWJjzbMP5bYCGXqjqon'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 1, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '3082387bba74fcd24c9700593d10418152d53c97', 'timestamp': '2021-03-09_20:35:33', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2137'} 2021-03-12T02:26:30.529 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-03-12T02:26:30.564 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-03-12T02:26:30.578 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/run/ceph 2021-03-12T02:26:30.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/run/ceph 2021-03-12T02:26:30.633 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2021-03-12T02:26:30.634 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2021-03-12T02:26:30.673 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-12T02:26:30.673 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-12T02:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-12T02:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-12T02:26:30.721 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 340 Links: 1 2021-03-12T02:26:30.722 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-12T02:26:30.722 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2021-03-12 02:23:52.784977441 +0000 2021-03-12T02:26:30.722 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2021-03-12 02:23:52.780977441 +0000 2021-03-12T02:26:30.722 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2021-03-12 02:23:52.780977441 +0000 2021-03-12T02:26:30.723 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2021-03-12T02:26:30.730 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2021-03-12T02:26:30.730 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2021-03-12T02:26:30.731 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000377623 s, 1.4 MB/s 2021-03-12T02:26:30.735 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-12T02:26:30.784 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-12T02:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-12T02:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 347 Links: 1 2021-03-12T02:26:30.785 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-12T02:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2021-03-12 02:23:53.160977433 +0000 2021-03-12T02:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2021-03-12 02:23:53.032977436 +0000 2021-03-12T02:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2021-03-12 02:23:53.032977436 +0000 2021-03-12T02:26:30.786 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2021-03-12T02:26:30.793 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2021-03-12T02:26:30.794 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2021-03-12T02:26:30.794 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000332786 s, 1.5 MB/s 2021-03-12T02:26:30.798 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-12T02:26:30.848 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-12T02:26:30.849 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-12T02:26:30.849 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 353 Links: 1 2021-03-12T02:26:30.849 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-12T02:26:30.849 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2021-03-12 02:23:53.296977430 +0000 2021-03-12T02:26:30.850 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2021-03-12 02:23:53.292977430 +0000 2021-03-12T02:26:30.850 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2021-03-12 02:23:53.292977430 +0000 2021-03-12T02:26:30.850 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2021-03-12T02:26:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2021-03-12T02:26:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2021-03-12T02:26:30.858 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000339834 s, 1.5 MB/s 2021-03-12T02:26:30.862 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-12T02:26:30.912 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-12T02:26:30.913 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-12T02:26:30.913 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 359 Links: 1 2021-03-12T02:26:30.913 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-12T02:26:30.913 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2021-03-12 02:23:53.648977422 +0000 2021-03-12T02:26:30.913 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2021-03-12 02:23:53.532977425 +0000 2021-03-12T02:26:30.914 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2021-03-12 02:23:53.532977425 +0000 2021-03-12T02:26:30.914 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2021-03-12T02:26:30.922 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2021-03-12T02:26:30.922 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2021-03-12T02:26:30.922 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000396304 s, 1.3 MB/s 2021-03-12T02:26:30.926 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-12T02:26:30.927 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-03-12T02:26:30.927 INFO:tasks.ceph:Generating config... 2021-03-12T02:26:30.929 INFO:tasks.ceph:[client] debug rgw = 20 2021-03-12T02:26:30.930 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-03-12T02:26:30.930 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-03-12T02:26:30.930 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-12T02:26:30.931 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-03-12T02:26:30.931 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-03-12T02:26:30.931 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-03-12T02:26:30.931 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-03-12T02:26:30.932 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-03-12T02:26:30.932 INFO:tasks.ceph:[client] setgroup = ceph 2021-03-12T02:26:30.932 INFO:tasks.ceph:[client] setuser = ceph 2021-03-12T02:26:30.933 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2021-03-12T02:26:30.933 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-03-12T02:26:30.933 INFO:tasks.ceph:[mgr] debug ms = 1 2021-03-12T02:26:30.933 INFO:tasks.ceph:[mon] debug mon = 20 2021-03-12T02:26:30.934 INFO:tasks.ceph:[mon] debug ms = 1 2021-03-12T02:26:30.934 INFO:tasks.ceph:[mon] debug paxos = 20 2021-03-12T02:26:30.934 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2021-03-12T02:26:30.935 INFO:tasks.ceph:[osd] debug filestore = 20 2021-03-12T02:26:30.935 INFO:tasks.ceph:[osd] debug journal = 20 2021-03-12T02:26:30.935 INFO:tasks.ceph:[osd] debug ms = 1 2021-03-12T02:26:30.935 INFO:tasks.ceph:[osd] debug osd = 25 2021-03-12T02:26:30.936 INFO:tasks.ceph:[osd] osd fast shutdown = False 2021-03-12T02:26:30.936 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-03-12T02:26:30.936 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-03-12T02:26:30.937 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-03-12T02:26:30.938 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-03-12T02:26:30.938 INFO:tasks.ceph:Setting up c1.mon.a... 2021-03-12T02:26:30.939 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2021-03-12T02:26:31.001 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/c1.keyring 2021-03-12T02:26:31.003 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2021-03-12T02:26:31.080 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-03-12T02:26:31.136 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.77')] 2021-03-12T02:26:31.137 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.77 --print /home/ubuntu/cephtest/c1.monmap 2021-03-12T02:26:31.205 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2021-03-12T02:26:31.205 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: generated fsid 542f5789-6a6d-4a4f-aeda-40ba0fd1f9ff 2021-03-12T02:26:31.206 INFO:teuthology.orchestra.run.smithi077.stdout:epoch 0 2021-03-12T02:26:31.206 INFO:teuthology.orchestra.run.smithi077.stdout:fsid 542f5789-6a6d-4a4f-aeda-40ba0fd1f9ff 2021-03-12T02:26:31.206 INFO:teuthology.orchestra.run.smithi077.stdout:last_changed 2021-03-12T02:26:31.209204+0000 2021-03-12T02:26:31.206 INFO:teuthology.orchestra.run.smithi077.stdout:created 2021-03-12T02:26:31.209204+0000 2021-03-12T02:26:31.207 INFO:teuthology.orchestra.run.smithi077.stdout:min_mon_release 0 (unknown) 2021-03-12T02:26:31.207 INFO:teuthology.orchestra.run.smithi077.stdout:0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2021-03-12T02:26:31.207 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2021-03-12T02:26:31.208 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 542f5789-6a6d-4a4f-aeda-40ba0fd1f9ff... 2021-03-12T02:26:31.209 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-03-12T02:26:31.252 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-03-12T02:26:31.283 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2021-03-12T02:26:31.284 INFO:teuthology.orchestra.run.smithi114.stdout: chdir = "" 2021-03-12T02:26:31.284 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-12T02:26:31.284 INFO:teuthology.orchestra.run.smithi114.stdout: auth supported = cephx 2021-03-12T02:26:31.284 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.284 INFO:teuthology.orchestra.run.smithi114.stdout: filestore xattr use omap = true 2021-03-12T02:26:31.285 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.285 INFO:teuthology.orchestra.run.smithi114.stdout: mon clock drift allowed = 1.000 2021-03-12T02:26:31.285 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.285 INFO:teuthology.orchestra.run.smithi114.stdout: osd crush chooseleaf type = 0 2021-03-12T02:26:31.286 INFO:teuthology.orchestra.run.smithi114.stdout: auth debug = true 2021-03-12T02:26:31.286 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.286 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on old message = true 2021-03-12T02:26:31.286 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on bug = true 2021-03-12T02:26:31.286 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.287 INFO:teuthology.orchestra.run.smithi114.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-12T02:26:31.287 INFO:teuthology.orchestra.run.smithi114.stdout: mon pg warn max object skew = 0 2021-03-12T02:26:31.287 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.287 INFO:teuthology.orchestra.run.smithi114.stdout: # disable pg_autoscaler by default for new pools 2021-03-12T02:26:31.288 INFO:teuthology.orchestra.run.smithi114.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-12T02:26:31.288 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.288 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default size = 2 2021-03-12T02:26:31.288 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.288 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow primary affinity = true 2021-03-12T02:26:31.289 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow pg remap = true 2021-03-12T02:26:31.289 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on legacy crush tunables = false 2021-03-12T02:26:31.289 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on crush straw calc version zero = false 2021-03-12T02:26:31.290 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on no sortbitwise = false 2021-03-12T02:26:31.290 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on osd down out interval zero = false 2021-03-12T02:26:31.290 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on too few osds = false 2021-03-12T02:26:31.290 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-12T02:26:31.290 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-12T02:26:31.291 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.291 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-12T02:26:31.291 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.291 INFO:teuthology.orchestra.run.smithi114.stdout: osd default data pool replay window = 5 2021-03-12T02:26:31.292 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.292 INFO:teuthology.orchestra.run.smithi114.stdout: mon allow pool delete = true 2021-03-12T02:26:31.292 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.292 INFO:teuthology.orchestra.run.smithi114.stdout: mon cluster log file level = debug 2021-03-12T02:26:31.292 INFO:teuthology.orchestra.run.smithi114.stdout: debug asserts on shutdown = true 2021-03-12T02:26:31.293 INFO:teuthology.orchestra.run.smithi114.stdout: mon health detail to clog = false 2021-03-12T02:26:31.293 INFO:teuthology.orchestra.run.smithi114.stdout: mon host = 172.21.15.77 2021-03-12T02:26:31.293 INFO:teuthology.orchestra.run.smithi114.stdout: osd heartbeat grace = 40 2021-03-12T02:26:31.293 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = 542f5789-6a6d-4a4f-aeda-40ba0fd1f9ff 2021-03-12T02:26:31.293 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.294 INFO:teuthology.orchestra.run.smithi114.stdout:[osd] 2021-03-12T02:26:31.294 INFO:teuthology.orchestra.run.smithi114.stdout: osd journal size = 100 2021-03-12T02:26:31.294 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.294 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub load threshold = 5.0 2021-03-12T02:26:31.295 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub max interval = 600 2021-03-12T02:26:31.295 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.295 INFO:teuthology.orchestra.run.smithi114.stdout: osd recover clone overlap = true 2021-03-12T02:26:31.295 INFO:teuthology.orchestra.run.smithi114.stdout: osd recovery max chunk = 1048576 2021-03-12T02:26:31.295 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.296 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug shutdown = true 2021-03-12T02:26:31.296 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug op order = true 2021-03-12T02:26:31.296 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify stray on activate = true 2021-03-12T02:26:31.296 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.296 INFO:teuthology.orchestra.run.smithi114.stdout: osd open classes on start = true 2021-03-12T02:26:31.297 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug pg log writeout = true 2021-03-12T02:26:31.297 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.297 INFO:teuthology.orchestra.run.smithi114.stdout: osd deep scrub update digest min age = 30 2021-03-12T02:26:31.297 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.298 INFO:teuthology.orchestra.run.smithi114.stdout: osd map max advance = 10 2021-03-12T02:26:31.298 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.298 INFO:teuthology.orchestra.run.smithi114.stdout: journal zero on create = true 2021-03-12T02:26:31.298 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.298 INFO:teuthology.orchestra.run.smithi114.stdout: filestore ondisk finisher threads = 3 2021-03-12T02:26:31.299 INFO:teuthology.orchestra.run.smithi114.stdout: filestore apply finisher threads = 3 2021-03-12T02:26:31.299 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.299 INFO:teuthology.orchestra.run.smithi114.stdout: bdev debug aio = true 2021-03-12T02:26:31.299 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug misdirected ops = true 2021-03-12T02:26:31.299 INFO:teuthology.orchestra.run.smithi114.stdout: debug filestore = 20 2021-03-12T02:26:31.300 INFO:teuthology.orchestra.run.smithi114.stdout: debug journal = 20 2021-03-12T02:26:31.300 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2021-03-12T02:26:31.300 INFO:teuthology.orchestra.run.smithi114.stdout: debug osd = 25 2021-03-12T02:26:31.300 INFO:teuthology.orchestra.run.smithi114.stdout: osd fast shutdown = False 2021-03-12T02:26:31.300 INFO:teuthology.orchestra.run.smithi114.stdout: rocksdb delete range threshold = 0 2021-03-12T02:26:31.301 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.301 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr] 2021-03-12T02:26:31.301 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2021-03-12T02:26:31.301 INFO:teuthology.orchestra.run.smithi114.stdout: debug mgr = 20 2021-03-12T02:26:31.302 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2021-03-12T02:26:31.302 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2021-03-12T02:26:31.302 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2021-03-12T02:26:31.302 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2021-03-12T02:26:31.303 INFO:teuthology.orchestra.run.smithi114.stdout: mgr/telemetry/nag = false 2021-03-12T02:26:31.303 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.303 INFO:teuthology.orchestra.run.smithi114.stdout:[mon] 2021-03-12T02:26:31.303 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2021-03-12T02:26:31.304 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2021-03-12T02:26:31.304 INFO:teuthology.orchestra.run.smithi114.stdout: debug paxos = 20 2021-03-12T02:26:31.304 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2021-03-12T02:26:31.304 INFO:teuthology.orchestra.run.smithi114.stdout: mon data avail warn = 5 2021-03-12T02:26:31.304 INFO:teuthology.orchestra.run.smithi114.stdout: mon mgr mkfs grace = 240 2021-03-12T02:26:31.305 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2021-03-12T02:26:31.305 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd reporter subtree level = osd 2021-03-12T02:26:31.305 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd prime pg temp = true 2021-03-12T02:26:31.305 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2021-03-12T02:26:31.305 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd crush smoke test = False 2021-03-12T02:26:31.306 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:26:31.306 INFO:teuthology.orchestra.run.smithi114.stdout:[client] 2021-03-12T02:26:31.306 INFO:teuthology.orchestra.run.smithi114.stdout: rgw cache enabled = true 2021-03-12T02:26:31.306 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable ops log = true 2021-03-12T02:26:31.307 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable usage log = true 2021-03-12T02:26:31.307 INFO:teuthology.orchestra.run.smithi114.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-12T02:26:31.307 INFO:teuthology.orchestra.run.smithi114.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-12T02:26:31.307 INFO:teuthology.orchestra.run.smithi114.stdout: debug rgw = 20 2021-03-12T02:26:31.307 INFO:teuthology.orchestra.run.smithi114.stdout: rgw crypt require ssl = False 2021-03-12T02:26:31.308 INFO:teuthology.orchestra.run.smithi114.stdout: rgw crypt s3 kms backend = testing 2021-03-12T02:26:31.308 INFO:teuthology.orchestra.run.smithi114.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-12T02:26:31.308 INFO:teuthology.orchestra.run.smithi114.stdout: rgw curl low speed time = 300 2021-03-12T02:26:31.308 INFO:teuthology.orchestra.run.smithi114.stdout: rgw data log num shards = 4 2021-03-12T02:26:31.308 INFO:teuthology.orchestra.run.smithi114.stdout: rgw md log max shards = 4 2021-03-12T02:26:31.309 INFO:teuthology.orchestra.run.smithi114.stdout: rgw sync log trim interval = 0 2021-03-12T02:26:31.309 INFO:teuthology.orchestra.run.smithi114.stdout: rgw sync obj etag verify = True 2021-03-12T02:26:31.309 INFO:teuthology.orchestra.run.smithi114.stdout: setgroup = ceph 2021-03-12T02:26:31.309 INFO:teuthology.orchestra.run.smithi114.stdout: setuser = ceph 2021-03-12T02:26:31.309 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.a] 2021-03-12T02:26:31.310 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.0] 2021-03-12T02:26:31.310 INFO:teuthology.orchestra.run.smithi114.stdout: osd_max_omap_entries_per_request = 10 2021-03-12T02:26:31.310 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.1] 2021-03-12T02:26:31.310 INFO:teuthology.orchestra.run.smithi114.stdout: osd_max_omap_entries_per_request = 1000 2021-03-12T02:26:31.311 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.2] 2021-03-12T02:26:31.311 INFO:teuthology.orchestra.run.smithi114.stdout: osd_max_omap_entries_per_request = 10000 2021-03-12T02:26:31.313 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2021-03-12T02:26:31.313 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2021-03-12T02:26:31.313 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-12T02:26:31.313 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2021-03-12T02:26:31.314 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.314 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2021-03-12T02:26:31.314 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.314 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2021-03-12T02:26:31.315 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.315 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2021-03-12T02:26:31.315 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2021-03-12T02:26:31.315 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.316 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2021-03-12T02:26:31.316 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2021-03-12T02:26:31.316 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.316 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-12T02:26:31.316 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2021-03-12T02:26:31.317 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.317 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2021-03-12T02:26:31.317 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-12T02:26:31.317 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.317 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2021-03-12T02:26:31.318 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.318 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2021-03-12T02:26:31.318 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2021-03-12T02:26:31.318 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2021-03-12T02:26:31.319 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2021-03-12T02:26:31.319 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2021-03-12T02:26:31.319 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2021-03-12T02:26:31.319 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2021-03-12T02:26:31.319 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-12T02:26:31.320 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-12T02:26:31.320 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.320 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-12T02:26:31.321 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.321 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2021-03-12T02:26:31.321 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.321 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2021-03-12T02:26:31.321 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.322 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2021-03-12T02:26:31.322 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2021-03-12T02:26:31.322 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2021-03-12T02:26:31.322 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = 172.21.15.77 2021-03-12T02:26:31.322 INFO:teuthology.orchestra.run.smithi077.stdout: osd heartbeat grace = 40 2021-03-12T02:26:31.323 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = 542f5789-6a6d-4a4f-aeda-40ba0fd1f9ff 2021-03-12T02:26:31.323 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.323 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2021-03-12T02:26:31.323 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2021-03-12T02:26:31.324 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.324 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2021-03-12T02:26:31.324 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2021-03-12T02:26:31.324 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.324 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2021-03-12T02:26:31.325 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2021-03-12T02:26:31.325 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.325 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2021-03-12T02:26:31.325 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2021-03-12T02:26:31.326 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2021-03-12T02:26:31.326 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.326 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2021-03-12T02:26:31.326 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2021-03-12T02:26:31.327 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.327 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2021-03-12T02:26:31.327 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.327 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2021-03-12T02:26:31.328 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.328 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2021-03-12T02:26:31.328 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.328 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2021-03-12T02:26:31.328 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2021-03-12T02:26:31.329 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.329 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2021-03-12T02:26:31.329 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2021-03-12T02:26:31.329 INFO:teuthology.orchestra.run.smithi077.stdout: debug filestore = 20 2021-03-12T02:26:31.329 INFO:teuthology.orchestra.run.smithi077.stdout: debug journal = 20 2021-03-12T02:26:31.330 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2021-03-12T02:26:31.330 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 25 2021-03-12T02:26:31.330 INFO:teuthology.orchestra.run.smithi077.stdout: osd fast shutdown = False 2021-03-12T02:26:31.331 INFO:teuthology.orchestra.run.smithi077.stdout: rocksdb delete range threshold = 0 2021-03-12T02:26:31.331 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.331 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2021-03-12T02:26:31.331 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2021-03-12T02:26:31.331 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2021-03-12T02:26:31.332 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2021-03-12T02:26:31.332 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2021-03-12T02:26:31.332 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2021-03-12T02:26:31.332 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2021-03-12T02:26:31.333 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2021-03-12T02:26:31.333 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.333 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2021-03-12T02:26:31.333 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2021-03-12T02:26:31.333 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2021-03-12T02:26:31.334 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2021-03-12T02:26:31.334 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2021-03-12T02:26:31.334 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2021-03-12T02:26:31.334 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2021-03-12T02:26:31.335 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2021-03-12T02:26:31.335 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2021-03-12T02:26:31.335 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2021-03-12T02:26:31.335 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2021-03-12T02:26:31.335 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd crush smoke test = False 2021-03-12T02:26:31.336 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:31.336 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2021-03-12T02:26:31.336 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2021-03-12T02:26:31.336 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2021-03-12T02:26:31.337 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2021-03-12T02:26:31.337 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-12T02:26:31.337 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-12T02:26:31.337 INFO:teuthology.orchestra.run.smithi077.stdout: debug rgw = 20 2021-03-12T02:26:31.337 INFO:teuthology.orchestra.run.smithi077.stdout: rgw crypt require ssl = False 2021-03-12T02:26:31.338 INFO:teuthology.orchestra.run.smithi077.stdout: rgw crypt s3 kms backend = testing 2021-03-12T02:26:31.338 INFO:teuthology.orchestra.run.smithi077.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-12T02:26:31.338 INFO:teuthology.orchestra.run.smithi077.stdout: rgw curl low speed time = 300 2021-03-12T02:26:31.338 INFO:teuthology.orchestra.run.smithi077.stdout: rgw data log num shards = 4 2021-03-12T02:26:31.339 INFO:teuthology.orchestra.run.smithi077.stdout: rgw md log max shards = 4 2021-03-12T02:26:31.339 INFO:teuthology.orchestra.run.smithi077.stdout: rgw sync log trim interval = 0 2021-03-12T02:26:31.339 INFO:teuthology.orchestra.run.smithi077.stdout: rgw sync obj etag verify = True 2021-03-12T02:26:31.339 INFO:teuthology.orchestra.run.smithi077.stdout: setgroup = ceph 2021-03-12T02:26:31.339 INFO:teuthology.orchestra.run.smithi077.stdout: setuser = ceph 2021-03-12T02:26:31.340 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2021-03-12T02:26:31.340 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.0] 2021-03-12T02:26:31.340 INFO:teuthology.orchestra.run.smithi077.stdout: osd_max_omap_entries_per_request = 10 2021-03-12T02:26:31.340 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.1] 2021-03-12T02:26:31.340 INFO:teuthology.orchestra.run.smithi077.stdout: osd_max_omap_entries_per_request = 1000 2021-03-12T02:26:31.341 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.2] 2021-03-12T02:26:31.341 INFO:teuthology.orchestra.run.smithi077.stdout: osd_max_omap_entries_per_request = 10000 2021-03-12T02:26:31.341 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2021-03-12T02:26:31.342 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2021-03-12T02:26:31.399 INFO:tasks.ceph:Copying monmap to all nodes... 2021-03-12T02:26:31.408 DEBUG:teuthology.orchestra.remote:smithi077:/etc/ceph/c1.keyring is 206B 2021-03-12T02:26:31.427 DEBUG:teuthology.orchestra.remote:smithi077:/home/ubuntu/cephtest/c1.monmap is 179B 2021-03-12T02:26:31.479 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi077.front.sepia.ceph.com 2021-03-12T02:26:31.480 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2021-03-12T02:26:31.480 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/c1.keyring 2021-03-12T02:26:31.481 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-03-12T02:26:31.505 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2021-03-12T02:26:31.505 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-03-12T02:26:31.556 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi114.front.sepia.ceph.com 2021-03-12T02:26:31.557 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2021-03-12T02:26:31.557 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/c1.keyring 2021-03-12T02:26:31.558 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-03-12T02:26:31.583 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2021-03-12T02:26:31.584 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-03-12T02:26:31.637 INFO:tasks.ceph:Setting up mon nodes... 2021-03-12T02:26:31.638 INFO:tasks.ceph:Setting up mgr nodes... 2021-03-12T02:26:31.638 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2021-03-12T02:26:31.680 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2021-03-12T02:26:31.683 INFO:tasks.ceph:Setting up mds nodes... 2021-03-12T02:26:31.683 INFO:tasks.ceph_client:Setting up client nodes... 2021-03-12T02:26:31.684 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2021-03-12T02:26:31.759 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/c1.client.0.keyring 2021-03-12T02:26:31.770 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2021-03-12T02:26:31.847 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/c1.client.1.keyring 2021-03-12T02:26:31.859 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-03-12T02:26:31.859 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-03-12T02:26:31.860 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2021-03-12T02:26:31.915 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-12T02:26:31.916 INFO:tasks.ceph:role: c1.osd.0 2021-03-12T02:26:31.916 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com 2021-03-12T02:26:31.916 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-03-12T02:26:32.435 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-03-12T02:26:32.435 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-12T02:26:32.436 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-12T02:26:32.436 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-12T02:26:32.436 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2021-03-12T02:26:32.436 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-12T02:26:32.436 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-12T02:26:32.437 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-12T02:26:32.437 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-12T02:26:32.438 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2021-03-12T02:26:32.438 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2021-03-12T02:26:32.510 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2021-03-12T02:26:32.521 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: /sbin/restorecon: command not found 2021-03-12T02:26:32.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-12T02:26:32.522 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2021-03-12T02:26:32.580 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-12T02:26:32.581 INFO:tasks.ceph:role: c1.osd.1 2021-03-12T02:26:32.581 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com 2021-03-12T02:26:32.581 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-03-12T02:26:33.098 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-03-12T02:26:33.099 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-12T02:26:33.099 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-12T02:26:33.099 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-12T02:26:33.099 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2021-03-12T02:26:33.100 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-12T02:26:33.100 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-12T02:26:33.100 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-12T02:26:33.100 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-12T02:26:33.101 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2021-03-12T02:26:33.102 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2021-03-12T02:26:33.173 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2021-03-12T02:26:33.185 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: /sbin/restorecon: command not found 2021-03-12T02:26:33.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-12T02:26:33.187 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2021-03-12T02:26:33.245 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-12T02:26:33.245 INFO:tasks.ceph:role: c1.osd.2 2021-03-12T02:26:33.246 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com 2021-03-12T02:26:33.246 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-03-12T02:26:33.755 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-03-12T02:26:33.756 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-12T02:26:33.756 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-12T02:26:33.756 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-12T02:26:33.756 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2021-03-12T02:26:33.757 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-12T02:26:33.757 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-12T02:26:33.757 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-12T02:26:33.757 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-12T02:26:33.759 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2021-03-12T02:26:33.759 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2021-03-12T02:26:33.804 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2021-03-12T02:26:33.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-12T02:26:33.859 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: /sbin/restorecon: command not found 2021-03-12T02:26:33.860 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2021-03-12T02:26:33.968 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:33.969+0000 7f3c9f672d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2021-03-12T02:26:33.969 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:33.969+0000 7f3c9f672d80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2021-03-12T02:26:33.969 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:33.969+0000 7f3c9f672d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-03-12T02:26:33.969 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:33.969+0000 7f3c9f672d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-03-12T02:26:33.969 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:33.969+0000 7f3c9f672d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-03-12T02:26:33.977 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:33.977+0000 7f3c9f672d80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2021-03-12T02:26:34.001 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:34.001+0000 7f3c9f672d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-12T02:26:35.872 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2021-03-12T02:26:35.889 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2021-03-12T02:26:35.966 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:35.965+0000 7f690534bd80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2021-03-12T02:26:35.966 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:35.965+0000 7f690534bd80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2021-03-12T02:26:35.966 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:35.965+0000 7f690534bd80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-03-12T02:26:35.967 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:35.965+0000 7f690534bd80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-03-12T02:26:35.967 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:35.965+0000 7f690534bd80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-03-12T02:26:35.968 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:35.969+0000 7f690534bd80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2021-03-12T02:26:35.983 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:35.981+0000 7f690534bd80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-12T02:26:37.876 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2021-03-12T02:26:37.934 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2021-03-12T02:26:38.016 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:38.017+0000 7f6754980d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2021-03-12T02:26:38.016 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:38.017+0000 7f6754980d80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2021-03-12T02:26:38.017 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:38.017+0000 7f6754980d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-03-12T02:26:38.017 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:38.017+0000 7f6754980d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-03-12T02:26:38.017 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:38.017+0000 7f6754980d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-03-12T02:26:38.017 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:38.017+0000 7f6754980d80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2021-03-12T02:26:38.033 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:26:38.033+0000 7f6754980d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-12T02:26:39.924 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2021-03-12T02:26:39.981 INFO:tasks.ceph:Reading keys from all nodes... 2021-03-12T02:26:39.982 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2021-03-12T02:26:40.028 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.qJ521IVEWU 2021-03-12T02:26:40.028 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /var/lib/ceph/mgr/c1-x/keyring /tmp/tmp.qJ521IVEWU 2021-03-12T02:26:40.089 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.qJ521IVEWU 2021-03-12T02:26:40.207 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.qJ521IVEWU is 56B 2021-03-12T02:26:40.295 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.qJ521IVEWU 2021-03-12T02:26:40.370 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2021-03-12T02:26:40.385 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.edDt4JExFQ 2021-03-12T02:26:40.385 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /var/lib/ceph/osd/c1-0/keyring /tmp/tmp.edDt4JExFQ 2021-03-12T02:26:40.451 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.edDt4JExFQ 2021-03-12T02:26:40.587 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.edDt4JExFQ is 56B 2021-03-12T02:26:40.624 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.edDt4JExFQ 2021-03-12T02:26:40.635 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2021-03-12T02:26:40.686 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.bApfve8mHd 2021-03-12T02:26:40.687 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /var/lib/ceph/osd/c1-1/keyring /tmp/tmp.bApfve8mHd 2021-03-12T02:26:40.746 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.bApfve8mHd 2021-03-12T02:26:40.864 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.bApfve8mHd is 56B 2021-03-12T02:26:40.892 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.bApfve8mHd 2021-03-12T02:26:40.899 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2021-03-12T02:26:40.951 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.3aJq3GG6xv 2021-03-12T02:26:40.952 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /var/lib/ceph/osd/c1-2/keyring /tmp/tmp.3aJq3GG6xv 2021-03-12T02:26:41.008 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.3aJq3GG6xv 2021-03-12T02:26:41.116 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.3aJq3GG6xv is 56B 2021-03-12T02:26:41.131 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.3aJq3GG6xv 2021-03-12T02:26:41.189 DEBUG:teuthology.orchestra.remote:smithi077:/etc/ceph/c1.client.0.keyring is 59B 2021-03-12T02:26:41.208 DEBUG:teuthology.orchestra.remote:smithi077:/etc/ceph/c1.client.1.keyring is 59B 2021-03-12T02:26:41.218 INFO:tasks.ceph:Adding keys to all mons... 2021-03-12T02:26:41.219 DEBUG:teuthology.orchestra.run.smithi077:> sudo tee -a /etc/ceph/c1.keyring 2021-03-12T02:26:41.231 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr.x] 2021-03-12T02:26:41.231 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBX0UpgksvRKBAABFU4EtH0qIrUZiTC3R9kjg== 2021-03-12T02:26:41.232 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.0] 2021-03-12T02:26:41.232 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBZ0Upg1jPuORAAReOJSwG9u2l5jOAJxCXDxA== 2021-03-12T02:26:41.232 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.1] 2021-03-12T02:26:41.232 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBb0UpgdZfJORAAqa6zj45mZHLiFWpAcWTcGw== 2021-03-12T02:26:41.232 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.2] 2021-03-12T02:26:41.233 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBe0UpgYHQwARAATaB8DEfs9KNlqvlf2cv5zQ== 2021-03-12T02:26:41.233 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2021-03-12T02:26:41.233 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBX0UpgLeJ7LRAAQwfvqRxFvOqZ9YWIMqUG0w== 2021-03-12T02:26:41.233 INFO:teuthology.orchestra.run.smithi077.stdout:[client.1] 2021-03-12T02:26:41.234 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQBX0Upg23PCMhAAm46bO/8h4Ir5xDnmG7ubhw== 2021-03-12T02:26:41.235 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-12T02:26:41.320 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-12T02:26:41.403 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-12T02:26:41.488 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-12T02:26:41.571 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-12T02:26:41.660 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-12T02:26:41.746 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-03-12T02:26:41.747 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2021-03-12T02:26:41.806 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2021-03-12T02:26:41.909 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2021-03-12T02:26:41.923 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/c1.monmap 2021-03-12T02:26:41.971 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2021-03-12T02:26:41.972 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2021-03-12T02:26:41.973 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2021-03-12T02:26:42.016 INFO:tasks.ceph.c1.mon.a:Started 2021-03-12T02:26:42.016 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2021-03-12T02:26:42.017 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2021-03-12T02:26:42.017 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2021-03-12T02:26:42.020 INFO:tasks.ceph.c1.mgr.x:Started 2021-03-12T02:26:42.020 INFO:tasks.ceph:Setting crush tunables to default 2021-03-12T02:26:42.021 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster c1 osd crush tunables default 2021-03-12T02:26:42.366 INFO:teuthology.orchestra.run.smithi077.stderr:adjusted tunables profile to default 2021-03-12T02:26:42.378 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2021-03-12T02:26:42.378 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2021-03-12T02:26:42.384 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.GVxWaj8DxA 2021-03-12T02:26:42.385 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /var/lib/ceph/osd/c1-0/fsid /tmp/tmp.GVxWaj8DxA 2021-03-12T02:26:42.442 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.GVxWaj8DxA 2021-03-12T02:26:42.555 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.GVxWaj8DxA is 37B 2021-03-12T02:26:42.569 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.GVxWaj8DxA 2021-03-12T02:26:42.576 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2021-03-12T02:26:42.623 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.6Ozr4fSFcx 2021-03-12T02:26:42.624 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /var/lib/ceph/osd/c1-1/fsid /tmp/tmp.6Ozr4fSFcx 2021-03-12T02:26:42.680 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.6Ozr4fSFcx 2021-03-12T02:26:42.793 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.6Ozr4fSFcx is 37B 2021-03-12T02:26:42.806 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.6Ozr4fSFcx 2021-03-12T02:26:42.813 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2021-03-12T02:26:42.862 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.K5klx1y3TC 2021-03-12T02:26:42.863 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /var/lib/ceph/osd/c1-2/fsid /tmp/tmp.K5klx1y3TC 2021-03-12T02:26:42.921 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.K5klx1y3TC 2021-03-12T02:26:43.029 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.K5klx1y3TC is 37B 2021-03-12T02:26:43.080 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.K5klx1y3TC 2021-03-12T02:26:43.093 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster c1 osd new 56100d43-9441-4d62-9d25-42c26501b6e7 0 2021-03-12T02:26:43.328 INFO:teuthology.orchestra.run.smithi077.stdout:0 2021-03-12T02:26:43.344 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster c1 osd new ec068e1f-7603-4fc4-9f0c-332cc60387e0 1 2021-03-12T02:26:43.548 INFO:teuthology.orchestra.run.smithi077.stdout:1 2021-03-12T02:26:43.559 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster c1 osd new dfcf2c9f-cf0d-42be-90e4-5cec71e83107 2 2021-03-12T02:26:43.754 INFO:teuthology.orchestra.run.smithi077.stdout:2 2021-03-12T02:26:43.766 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2021-03-12T02:26:43.767 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2021-03-12T02:26:43.777 INFO:tasks.ceph.c1.osd.0:Started 2021-03-12T02:26:43.777 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2021-03-12T02:26:43.778 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2021-03-12T02:26:43.783 INFO:tasks.ceph.c1.osd.1:Started 2021-03-12T02:26:43.784 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2021-03-12T02:26:43.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2021-03-12T02:26:43.792 INFO:tasks.ceph.c1.osd.2:Started 2021-03-12T02:26:43.793 INFO:tasks.ceph:Waiting for OSDs to come up 2021-03-12T02:26:44.096 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-03-12T02:26:44.124 INFO:tasks.ceph.c1.osd.0.smithi077.stderr:2021-03-12T02:26:44.125+0000 7fca16241d80 -1 Falling back to public interface 2021-03-12T02:26:44.124 INFO:tasks.ceph.c1.osd.2.smithi077.stderr:2021-03-12T02:26:44.125+0000 7fe0e9e04d80 -1 Falling back to public interface 2021-03-12T02:26:44.269 INFO:teuthology.misc.health.smithi077.stdout: 2021-03-12T02:26:44.270 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":5,"fsid":"542f5789-6a6d-4a4f-aeda-40ba0fd1f9ff","created":"2021-03-12T02:26:42.161169+0000","modified":"2021-03-12T02:26:43.755978+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"56100d43-9441-4d62-9d25-42c26501b6e7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ec068e1f-7603-4fc4-9f0c-332cc60387e0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"dfcf2c9f-cf0d-42be-90e4-5cec71e83107","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-12T02:26:44.282 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-03-12T02:26:44.391 INFO:tasks.ceph.c1.osd.1.smithi077.stderr:2021-03-12T02:26:44.393+0000 7f455674ad80 -1 Falling back to public interface 2021-03-12T02:26:44.728 INFO:tasks.ceph.c1.osd.2.smithi077.stderr:2021-03-12T02:26:44.729+0000 7fe0e9e04d80 -1 osd.2 0 log_to_monitors {default=true} 2021-03-12T02:26:44.734 INFO:tasks.ceph.c1.osd.0.smithi077.stderr:2021-03-12T02:26:44.737+0000 7fca16241d80 -1 osd.0 0 log_to_monitors {default=true} 2021-03-12T02:26:44.985 INFO:tasks.ceph.c1.osd.1.smithi077.stderr:2021-03-12T02:26:44.985+0000 7f455674ad80 -1 osd.1 0 log_to_monitors {default=true} 2021-03-12T02:26:46.194 INFO:tasks.ceph.c1.osd.0.smithi077.stderr:2021-03-12T02:26:46.197+0000 7fc9ffc8d700 -1 osd.0 0 waiting for initial osdmap 2021-03-12T02:26:46.195 INFO:tasks.ceph.c1.osd.2.smithi077.stderr:2021-03-12T02:26:46.197+0000 7fe0d3850700 -1 osd.2 0 waiting for initial osdmap 2021-03-12T02:26:46.195 INFO:tasks.ceph.c1.osd.1.smithi077.stderr:2021-03-12T02:26:46.197+0000 7f4540196700 -1 osd.1 0 waiting for initial osdmap 2021-03-12T02:26:50.584 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-03-12T02:26:50.853 INFO:teuthology.misc.health.smithi077.stdout: 2021-03-12T02:26:50.854 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":10,"fsid":"542f5789-6a6d-4a4f-aeda-40ba0fd1f9ff","created":"2021-03-12T02:26:42.161169+0000","modified":"2021-03-12T02:26:49.209426+0000","last_up_change":"2021-03-12T02:26:47.193633+0000","last_in_change":"2021-03-12T02:26:47.193633+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-12T02:26:46.013644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"56100d43-9441-4d62-9d25-42c26501b6e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":13508},{"type":"v1","addr":"172.21.15.77:6801","nonce":13508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6803","nonce":13508},{"type":"v1","addr":"172.21.15.77:6805","nonce":13508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":13508},{"type":"v1","addr":"172.21.15.77:6812","nonce":13508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":13508},{"type":"v1","addr":"172.21.15.77:6808","nonce":13508}]},"public_addr":"172.21.15.77:6801/13508","cluster_addr":"172.21.15.77:6805/13508","heartbeat_back_addr":"172.21.15.77:6812/13508","heartbeat_front_addr":"172.21.15.77:6808/13508","state":["exists","up"]},{"osd":1,"uuid":"ec068e1f-7603-4fc4-9f0c-332cc60387e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":13509},{"type":"v1","addr":"172.21.15.77:6817","nonce":13509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":13509},{"type":"v1","addr":"172.21.15.77:6819","nonce":13509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":13509},{"type":"v1","addr":"172.21.15.77:6823","nonce":13509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":13509},{"type":"v1","addr":"172.21.15.77:6821","nonce":13509}]},"public_addr":"172.21.15.77:6817/13509","cluster_addr":"172.21.15.77:6819/13509","heartbeat_back_addr":"172.21.15.77:6823/13509","heartbeat_front_addr":"172.21.15.77:6821/13509","state":["exists","up"]},{"osd":2,"uuid":"dfcf2c9f-cf0d-42be-90e4-5cec71e83107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":13510},{"type":"v1","addr":"172.21.15.77:6804","nonce":13510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":13510},{"type":"v1","addr":"172.21.15.77:6809","nonce":13510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":13510},{"type":"v1","addr":"172.21.15.77:6815","nonce":13510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":13510},{"type":"v1","addr":"172.21.15.77:6813","nonce":13510}]},"public_addr":"172.21.15.77:6804/13510","cluster_addr":"172.21.15.77:6809/13510","heartbeat_back_addr":"172.21.15.77:6815/13510","heartbeat_front_addr":"172.21.15.77:6813/13510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:26:45.725432+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:26:45.975000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:26:45.703739+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-12T02:26:50.865 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-03-12T02:26:50.866 INFO:tasks.ceph:Creating RBD pool 2021-03-12T02:26:50.866 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster c1 osd pool create rbd 8 2021-03-12T02:26:51.234 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'rbd' created 2021-03-12T02:26:51.246 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-03-12T02:26:52.252 INFO:teuthology.orchestra.run.smithi077.stderr:enabled application 'rbd' on pool 'rbd' 2021-03-12T02:26:52.265 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2021-03-12T02:26:52.266 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2021-03-12T02:26:52.266 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-03-12T02:26:52.530 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:52.596 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":12,"fsid":"542f5789-6a6d-4a4f-aeda-40ba0fd1f9ff","created":"2021-03-12T02:26:42.161169+0000","modified":"2021-03-12T02:26:52.229494+0000","last_up_change":"2021-03-12T02:26:47.193633+0000","last_in_change":"2021-03-12T02:26:47.193633+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-12T02:26:46.013644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-12T02:26:51.143111+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"56100d43-9441-4d62-9d25-42c26501b6e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":13508},{"type":"v1","addr":"172.21.15.77:6801","nonce":13508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6803","nonce":13508},{"type":"v1","addr":"172.21.15.77:6805","nonce":13508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":13508},{"type":"v1","addr":"172.21.15.77:6812","nonce":13508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":13508},{"type":"v1","addr":"172.21.15.77:6808","nonce":13508}]},"public_addr":"172.21.15.77:6801/13508","cluster_addr":"172.21.15.77:6805/13508","heartbeat_back_addr":"172.21.15.77:6812/13508","heartbeat_front_addr":"172.21.15.77:6808/13508","state":["exists","up"]},{"osd":1,"uuid":"ec068e1f-7603-4fc4-9f0c-332cc60387e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":13509},{"type":"v1","addr":"172.21.15.77:6817","nonce":13509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":13509},{"type":"v1","addr":"172.21.15.77:6819","nonce":13509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":13509},{"type":"v1","addr":"172.21.15.77:6823","nonce":13509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":13509},{"type":"v1","addr":"172.21.15.77:6821","nonce":13509}]},"public_addr":"172.21.15.77:6817/13509","cluster_addr":"172.21.15.77:6819/13509","heartbeat_back_addr":"172.21.15.77:6823/13509","heartbeat_front_addr":"172.21.15.77:6821/13509","state":["exists","up"]},{"osd":2,"uuid":"dfcf2c9f-cf0d-42be-90e4-5cec71e83107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":13510},{"type":"v1","addr":"172.21.15.77:6804","nonce":13510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":13510},{"type":"v1","addr":"172.21.15.77:6809","nonce":13510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":13510},{"type":"v1","addr":"172.21.15.77:6815","nonce":13510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":13510},{"type":"v1","addr":"172.21.15.77:6813","nonce":13510}]},"public_addr":"172.21.15.77:6804/13510","cluster_addr":"172.21.15.77:6809/13510","heartbeat_back_addr":"172.21.15.77:6815/13510","heartbeat_front_addr":"172.21.15.77:6813/13510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:26:45.725432+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:26:45.975000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:26:45.703739+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-12T02:26:52.597 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-12T02:26:46.013644+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-03-12T02:26:51.143111+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-03-12T02:26:52.598 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get device_health_metrics pg_num 2021-03-12T02:26:52.877 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2021-03-12T02:26:52.896 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get rbd pg_num 2021-03-12T02:26:53.172 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 8 2021-03-12T02:26:53.432 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2021-03-12T02:26:53.433 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2021-03-12T02:26:53.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early mgr dump --format=json 2021-03-12T02:26:53.750 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:53.898 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":13299},{"type":"v1","addr":"172.21.15.77:6825","nonce":13299}]},"active_addr":"172.21.15.77:6825/13299","active_change":"2021-03-12T02:26:45.964194+0000","active_mgr_features":4540138292840890367,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":1356572122}]},{"addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":3872465302}]},{"addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":234420128}]}]}} 2021-03-12T02:26:53.901 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2021-03-12T02:26:53.901 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2021-03-12T02:26:53.902 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2021-03-12T02:26:54.174 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:54.175 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"fsid":"542f5789-6a6d-4a4f-aeda-40ba0fd1f9ff","created":"2021-03-12T02:26:42.161169+0000","modified":"2021-03-12T02:26:53.238926+0000","last_up_change":"2021-03-12T02:26:47.193633+0000","last_in_change":"2021-03-12T02:26:47.193633+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-12T02:26:46.013644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-12T02:26:51.143111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"56100d43-9441-4d62-9d25-42c26501b6e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":13508},{"type":"v1","addr":"172.21.15.77:6801","nonce":13508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6803","nonce":13508},{"type":"v1","addr":"172.21.15.77:6805","nonce":13508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":13508},{"type":"v1","addr":"172.21.15.77:6812","nonce":13508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":13508},{"type":"v1","addr":"172.21.15.77:6808","nonce":13508}]},"public_addr":"172.21.15.77:6801/13508","cluster_addr":"172.21.15.77:6805/13508","heartbeat_back_addr":"172.21.15.77:6812/13508","heartbeat_front_addr":"172.21.15.77:6808/13508","state":["exists","up"]},{"osd":1,"uuid":"ec068e1f-7603-4fc4-9f0c-332cc60387e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":13509},{"type":"v1","addr":"172.21.15.77:6817","nonce":13509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":13509},{"type":"v1","addr":"172.21.15.77:6819","nonce":13509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":13509},{"type":"v1","addr":"172.21.15.77:6823","nonce":13509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":13509},{"type":"v1","addr":"172.21.15.77:6821","nonce":13509}]},"public_addr":"172.21.15.77:6817/13509","cluster_addr":"172.21.15.77:6819/13509","heartbeat_back_addr":"172.21.15.77:6823/13509","heartbeat_front_addr":"172.21.15.77:6821/13509","state":["exists","up"]},{"osd":2,"uuid":"dfcf2c9f-cf0d-42be-90e4-5cec71e83107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":13510},{"type":"v1","addr":"172.21.15.77:6804","nonce":13510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":13510},{"type":"v1","addr":"172.21.15.77:6809","nonce":13510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":13510},{"type":"v1","addr":"172.21.15.77:6815","nonce":13510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":13510},{"type":"v1","addr":"172.21.15.77:6813","nonce":13510}]},"public_addr":"172.21.15.77:6804/13510","cluster_addr":"172.21.15.77:6809/13510","heartbeat_back_addr":"172.21.15.77:6815/13510","heartbeat_front_addr":"172.21.15.77:6813/13510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:26:45.725432+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:26:45.975000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:26:45.703739+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-12T02:26:54.188 INFO:tasks.ceph.ceph_manager.c1:all up! 2021-03-12T02:26:54.188 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2021-03-12T02:26:54.457 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:54.610 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"fsid":"542f5789-6a6d-4a4f-aeda-40ba0fd1f9ff","created":"2021-03-12T02:26:42.161169+0000","modified":"2021-03-12T02:26:53.238926+0000","last_up_change":"2021-03-12T02:26:47.193633+0000","last_in_change":"2021-03-12T02:26:47.193633+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-12T02:26:46.013644+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-12T02:26:51.143111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"56100d43-9441-4d62-9d25-42c26501b6e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":13508},{"type":"v1","addr":"172.21.15.77:6801","nonce":13508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6803","nonce":13508},{"type":"v1","addr":"172.21.15.77:6805","nonce":13508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":13508},{"type":"v1","addr":"172.21.15.77:6812","nonce":13508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6807","nonce":13508},{"type":"v1","addr":"172.21.15.77:6808","nonce":13508}]},"public_addr":"172.21.15.77:6801/13508","cluster_addr":"172.21.15.77:6805/13508","heartbeat_back_addr":"172.21.15.77:6812/13508","heartbeat_front_addr":"172.21.15.77:6808/13508","state":["exists","up"]},{"osd":1,"uuid":"ec068e1f-7603-4fc4-9f0c-332cc60387e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":13509},{"type":"v1","addr":"172.21.15.77:6817","nonce":13509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":13509},{"type":"v1","addr":"172.21.15.77:6819","nonce":13509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":13509},{"type":"v1","addr":"172.21.15.77:6823","nonce":13509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":13509},{"type":"v1","addr":"172.21.15.77:6821","nonce":13509}]},"public_addr":"172.21.15.77:6817/13509","cluster_addr":"172.21.15.77:6819/13509","heartbeat_back_addr":"172.21.15.77:6823/13509","heartbeat_front_addr":"172.21.15.77:6821/13509","state":["exists","up"]},{"osd":2,"uuid":"dfcf2c9f-cf0d-42be-90e4-5cec71e83107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":13510},{"type":"v1","addr":"172.21.15.77:6804","nonce":13510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":13510},{"type":"v1","addr":"172.21.15.77:6809","nonce":13510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":13510},{"type":"v1","addr":"172.21.15.77:6815","nonce":13510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6811","nonce":13510},{"type":"v1","addr":"172.21.15.77:6813","nonce":13510}]},"public_addr":"172.21.15.77:6804/13510","cluster_addr":"172.21.15.77:6809/13510","heartbeat_back_addr":"172.21.15.77:6815/13510","heartbeat_front_addr":"172.21.15.77:6813/13510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:26:45.725432+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:26:45.975000+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:26:45.703739+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-12T02:26:54.612 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.0 flush_pg_stats 2021-03-12T02:26:54.768 INFO:teuthology.orchestra.run.smithi077.stdout:34359738371 2021-03-12T02:26:54.768 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.1 flush_pg_stats 2021-03-12T02:26:54.924 INFO:teuthology.orchestra.run.smithi077.stdout:34359738372 2021-03-12T02:26:55.183 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.2 flush_pg_stats 2021-03-12T02:26:55.352 INFO:teuthology.orchestra.run.smithi077.stdout:34359738371 2021-03-12T02:26:55.353 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2021-03-12T02:26:55.628 INFO:teuthology.orchestra.run.smithi077.stdout:34359738370 2021-03-12T02:26:57.098 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2021-03-12T02:26:58.100 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2021-03-12T02:26:58.383 INFO:teuthology.orchestra.run.smithi077.stdout:34359738371 2021-03-12T02:26:58.396 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.0 2021-03-12T02:26:58.397 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.1 2021-03-12T02:26:58.669 INFO:teuthology.orchestra.run.smithi077.stdout:34359738372 2021-03-12T02:26:58.682 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.1 2021-03-12T02:26:58.683 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.2 2021-03-12T02:26:58.945 INFO:teuthology.orchestra.run.smithi077.stdout:34359738371 2021-03-12T02:26:58.959 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.2 2021-03-12T02:26:58.959 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2021-03-12T02:26:58.960 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2021-03-12T02:26:59.234 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:59.236 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2021-03-12T02:26:59.247 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-03-12T02:26:57.974854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.733454"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.263016+0000","last_change":"2021-03-12T02:26:52.261856+0000","last_active":"2021-03-12T02:26:52.263016+0000","last_peered":"2021-03-12T02:26:52.263016+0000","last_clean":"2021-03-12T02:26:52.263016+0000","last_became_active":"2021-03-12T02:26:52.261532+0000","last_became_peered":"2021-03-12T02:26:52.261532+0000","last_unstale":"2021-03-12T02:26:52.263016+0000","last_undegraded":"2021-03-12T02:26:52.263016+0000","last_fullsized":"2021-03-12T02:26:52.263016+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_clean_scrub_stamp":"2021-03-12T02:26:51.222613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.262667+0000","last_change":"2021-03-12T02:26:52.260619+0000","last_active":"2021-03-12T02:26:52.262667+0000","last_peered":"2021-03-12T02:26:52.262667+0000","last_clean":"2021-03-12T02:26:52.262667+0000","last_became_active":"2021-03-12T02:26:52.260211+0000","last_became_peered":"2021-03-12T02:26:52.260211+0000","last_unstale":"2021-03-12T02:26:52.262667+0000","last_undegraded":"2021-03-12T02:26:52.262667+0000","last_fullsized":"2021-03-12T02:26:52.262667+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_clean_scrub_stamp":"2021-03-12T02:26:51.222613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.260339+0000","last_change":"2021-03-12T02:26:52.255525+0000","last_active":"2021-03-12T02:26:52.260339+0000","last_peered":"2021-03-12T02:26:52.260339+0000","last_clean":"2021-03-12T02:26:52.260339+0000","last_became_active":"2021-03-12T02:26:52.255013+0000","last_became_peered":"2021-03-12T02:26:52.255013+0000","last_unstale":"2021-03-12T02:26:52.260339+0000","last_undegraded":"2021-03-12T02:26:52.260339+0000","last_fullsized":"2021-03-12T02:26:52.260339+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_clean_scrub_stamp":"2021-03-12T02:26:51.222613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.252944+0000","last_change":"2021-03-12T02:26:52.252944+0000","last_active":"2021-03-12T02:26:52.252944+0000","last_peered":"2021-03-12T02:26:52.252944+0000","last_clean":"2021-03-12T02:26:52.252944+0000","last_became_active":"2021-03-12T02:26:52.252614+0000","last_became_peered":"2021-03-12T02:26:52.252614+0000","last_unstale":"2021-03-12T02:26:52.252944+0000","last_undegraded":"2021-03-12T02:26:52.252944+0000","last_fullsized":"2021-03-12T02:26:52.252944+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_clean_scrub_stamp":"2021-03-12T02:26:51.222613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.250003+0000","last_change":"2021-03-12T02:26:48.792596+0000","last_active":"2021-03-12T02:26:52.250003+0000","last_peered":"2021-03-12T02:26:52.250003+0000","last_clean":"2021-03-12T02:26:52.250003+0000","last_became_active":"2021-03-12T02:26:48.791168+0000","last_became_peered":"2021-03-12T02:26:48.791168+0000","last_unstale":"2021-03-12T02:26:52.250003+0000","last_undegraded":"2021-03-12T02:26:52.250003+0000","last_fullsized":"2021-03-12T02:26:52.250003+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:46.182555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:46.182555+0000","last_clean_scrub_stamp":"2021-03-12T02:26:46.182555+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.260468+0000","last_change":"2021-03-12T02:26:52.256018+0000","last_active":"2021-03-12T02:26:52.260468+0000","last_peered":"2021-03-12T02:26:52.260468+0000","last_clean":"2021-03-12T02:26:52.260468+0000","last_became_active":"2021-03-12T02:26:52.255265+0000","last_became_peered":"2021-03-12T02:26:52.255265+0000","last_unstale":"2021-03-12T02:26:52.260468+0000","last_undegraded":"2021-03-12T02:26:52.260468+0000","last_fullsized":"2021-03-12T02:26:52.260468+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_clean_scrub_stamp":"2021-03-12T02:26:51.222613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.253548+0000","last_change":"2021-03-12T02:26:52.253548+0000","last_active":"2021-03-12T02:26:52.253548+0000","last_peered":"2021-03-12T02:26:52.253548+0000","last_clean":"2021-03-12T02:26:52.253548+0000","last_became_active":"2021-03-12T02:26:52.253158+0000","last_became_peered":"2021-03-12T02:26:52.253158+0000","last_unstale":"2021-03-12T02:26:52.253548+0000","last_undegraded":"2021-03-12T02:26:52.253548+0000","last_fullsized":"2021-03-12T02:26:52.253548+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_clean_scrub_stamp":"2021-03-12T02:26:51.222613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.255570+0000","last_change":"2021-03-12T02:26:52.255570+0000","last_active":"2021-03-12T02:26:52.255570+0000","last_peered":"2021-03-12T02:26:52.255570+0000","last_clean":"2021-03-12T02:26:52.255570+0000","last_became_active":"2021-03-12T02:26:52.255239+0000","last_became_peered":"2021-03-12T02:26:52.255239+0000","last_unstale":"2021-03-12T02:26:52.255570+0000","last_undegraded":"2021-03-12T02:26:52.255570+0000","last_fullsized":"2021-03-12T02:26:52.255570+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_clean_scrub_stamp":"2021-03-12T02:26:51.222613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.261190+0000","last_change":"2021-03-12T02:26:52.260585+0000","last_active":"2021-03-12T02:26:52.261190+0000","last_peered":"2021-03-12T02:26:52.261190+0000","last_clean":"2021-03-12T02:26:52.261190+0000","last_became_active":"2021-03-12T02:26:52.260223+0000","last_became_peered":"2021-03-12T02:26:52.260223+0000","last_unstale":"2021-03-12T02:26:52.261190+0000","last_undegraded":"2021-03-12T02:26:52.261190+0000","last_fullsized":"2021-03-12T02:26:52.261190+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_clean_scrub_stamp":"2021-03-12T02:26:51.222613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]}]}} 2021-03-12T02:26:59.248 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2021-03-12T02:26:59.511 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:59.513 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2021-03-12T02:26:59.531 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-03-12T02:26:57.974854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.733454"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.263016+0000","last_change":"2021-03-12T02:26:52.261856+0000","last_active":"2021-03-12T02:26:52.263016+0000","last_peered":"2021-03-12T02:26:52.263016+0000","last_clean":"2021-03-12T02:26:52.263016+0000","last_became_active":"2021-03-12T02:26:52.261532+0000","last_became_peered":"2021-03-12T02:26:52.261532+0000","last_unstale":"2021-03-12T02:26:52.263016+0000","last_undegraded":"2021-03-12T02:26:52.263016+0000","last_fullsized":"2021-03-12T02:26:52.263016+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_clean_scrub_stamp":"2021-03-12T02:26:51.222613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.262667+0000","last_change":"2021-03-12T02:26:52.260619+0000","last_active":"2021-03-12T02:26:52.262667+0000","last_peered":"2021-03-12T02:26:52.262667+0000","last_clean":"2021-03-12T02:26:52.262667+0000","last_became_active":"2021-03-12T02:26:52.260211+0000","last_became_peered":"2021-03-12T02:26:52.260211+0000","last_unstale":"2021-03-12T02:26:52.262667+0000","last_undegraded":"2021-03-12T02:26:52.262667+0000","last_fullsized":"2021-03-12T02:26:52.262667+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_clean_scrub_stamp":"2021-03-12T02:26:51.222613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.260339+0000","last_change":"2021-03-12T02:26:52.255525+0000","last_active":"2021-03-12T02:26:52.260339+0000","last_peered":"2021-03-12T02:26:52.260339+0000","last_clean":"2021-03-12T02:26:52.260339+0000","last_became_active":"2021-03-12T02:26:52.255013+0000","last_became_peered":"2021-03-12T02:26:52.255013+0000","last_unstale":"2021-03-12T02:26:52.260339+0000","last_undegraded":"2021-03-12T02:26:52.260339+0000","last_fullsized":"2021-03-12T02:26:52.260339+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_clean_scrub_stamp":"2021-03-12T02:26:51.222613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.252944+0000","last_change":"2021-03-12T02:26:52.252944+0000","last_active":"2021-03-12T02:26:52.252944+0000","last_peered":"2021-03-12T02:26:52.252944+0000","last_clean":"2021-03-12T02:26:52.252944+0000","last_became_active":"2021-03-12T02:26:52.252614+0000","last_became_peered":"2021-03-12T02:26:52.252614+0000","last_unstale":"2021-03-12T02:26:52.252944+0000","last_undegraded":"2021-03-12T02:26:52.252944+0000","last_fullsized":"2021-03-12T02:26:52.252944+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_clean_scrub_stamp":"2021-03-12T02:26:51.222613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.250003+0000","last_change":"2021-03-12T02:26:48.792596+0000","last_active":"2021-03-12T02:26:52.250003+0000","last_peered":"2021-03-12T02:26:52.250003+0000","last_clean":"2021-03-12T02:26:52.250003+0000","last_became_active":"2021-03-12T02:26:48.791168+0000","last_became_peered":"2021-03-12T02:26:48.791168+0000","last_unstale":"2021-03-12T02:26:52.250003+0000","last_undegraded":"2021-03-12T02:26:52.250003+0000","last_fullsized":"2021-03-12T02:26:52.250003+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:46.182555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:46.182555+0000","last_clean_scrub_stamp":"2021-03-12T02:26:46.182555+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.260468+0000","last_change":"2021-03-12T02:26:52.256018+0000","last_active":"2021-03-12T02:26:52.260468+0000","last_peered":"2021-03-12T02:26:52.260468+0000","last_clean":"2021-03-12T02:26:52.260468+0000","last_became_active":"2021-03-12T02:26:52.255265+0000","last_became_peered":"2021-03-12T02:26:52.255265+0000","last_unstale":"2021-03-12T02:26:52.260468+0000","last_undegraded":"2021-03-12T02:26:52.260468+0000","last_fullsized":"2021-03-12T02:26:52.260468+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_clean_scrub_stamp":"2021-03-12T02:26:51.222613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.253548+0000","last_change":"2021-03-12T02:26:52.253548+0000","last_active":"2021-03-12T02:26:52.253548+0000","last_peered":"2021-03-12T02:26:52.253548+0000","last_clean":"2021-03-12T02:26:52.253548+0000","last_became_active":"2021-03-12T02:26:52.253158+0000","last_became_peered":"2021-03-12T02:26:52.253158+0000","last_unstale":"2021-03-12T02:26:52.253548+0000","last_undegraded":"2021-03-12T02:26:52.253548+0000","last_fullsized":"2021-03-12T02:26:52.253548+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_clean_scrub_stamp":"2021-03-12T02:26:51.222613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.255570+0000","last_change":"2021-03-12T02:26:52.255570+0000","last_active":"2021-03-12T02:26:52.255570+0000","last_peered":"2021-03-12T02:26:52.255570+0000","last_clean":"2021-03-12T02:26:52.255570+0000","last_became_active":"2021-03-12T02:26:52.255239+0000","last_became_peered":"2021-03-12T02:26:52.255239+0000","last_unstale":"2021-03-12T02:26:52.255570+0000","last_undegraded":"2021-03-12T02:26:52.255570+0000","last_fullsized":"2021-03-12T02:26:52.255570+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_clean_scrub_stamp":"2021-03-12T02:26:51.222613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:26:52.261190+0000","last_change":"2021-03-12T02:26:52.260585+0000","last_active":"2021-03-12T02:26:52.261190+0000","last_peered":"2021-03-12T02:26:52.261190+0000","last_clean":"2021-03-12T02:26:52.261190+0000","last_became_active":"2021-03-12T02:26:52.260223+0000","last_became_peered":"2021-03-12T02:26:52.260223+0000","last_unstale":"2021-03-12T02:26:52.261190+0000","last_undegraded":"2021-03-12T02:26:52.261190+0000","last_fullsized":"2021-03-12T02:26:52.261190+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:26:51.222613+0000","last_clean_scrub_stamp":"2021-03-12T02:26:51.222613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]}]}]}} 2021-03-12T02:26:59.532 INFO:tasks.ceph.ceph_manager.c1:clean! 2021-03-12T02:26:59.533 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2021-03-12T02:26:59.533 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2021-03-12T02:26:59.533 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early health --format=json 2021-03-12T02:26:59.847 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:26:59.860 INFO:teuthology.orchestra.run.smithi077.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-03-12T02:26:59.861 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2021-03-12T02:26:59.861 INFO:teuthology.run_tasks:Running task ceph... 2021-03-12T02:26:59.870 INFO:tasks.ceph:Creating ceph cluster c2... 2021-03-12T02:26:59.870 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 1, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-03-12T02:26:59.871 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2021-03-09_20:35:33-rgw-wip-yuri4-testing-2021-03-09-1006-octopus-distro-basic-smithi/5951134', 'branch': 'wip-yuri4-testing-2021-03-09-1006-octopus', 'description': 'rgw/multisite/{clusters frontend/civetweb omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '5951134', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2021-03-09_20:35:33-rgw-wip-yuri4-testing-2021-03-09-1006-octopus-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2021-03-09-1006-octopus'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 1, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb'}}, 'rgw': {'compression type': 'random', 'frontend': 'civetweb'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-yuri4-testing-2021-03-09-1006-octopus', 'sha1': '3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-yuri4-testing-2021-03-09-1006-octopus', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb', 'targets': {'smithi077.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgWjnm/iuIIjyr4cN1NZVVqXGxv1uNGNsn9pMzqAsfjgBCcQPK/LuPILtOZjDQo5bI2ch8/+/Y+iRPbmDlNKZegCoe/E9U0ccVreM3hIU1iM+A7NQf59jE++pSt0z/9KX1ZeD5ValQ79L1wDholUHsR5CKNq8t9uTah2Jv9LzrSAFhIHWD7gsiUtQV5Nh+zxiXElh2GoC6maYlFqBiJBM/SGU/hX8DwhFnXS7e4E4NshtegWeijAt93Zsw4Anpu1ChfqYwB4uJW4EsbZmcx2zpgNbisXclfgtf/udwb4qJQRE8XjO4wWaqCGtNh9s4wFZpSaVQGBvDLqvsaF85L4SN', 'smithi114.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXjz7S6TEnBMVHdLOfCMhKgTnCP9W9q8QDBYiH63QhmXddO2kAhw1oD+J8nw92mrC6qbK66SSkhGRPGDRFzB7MiY1tw4fPWBuzrDC4WDnBFHuD2tOLcb9Lwoue6DlV+bzChQpW6RgRYscpZJpC99lSnLHxX3cI4vS3ScpDBrnGG4NfqFzdKE1Hs8TxOmV4PhqTZTEo08/V7/WKGS8zLfmKPm1/bPgbbl/cJ+xFfYc0ro8wd6aQmdK+eZJx7DYK13Lt27YHVLq/aeRBkuggXLYZHz4gQAwXwVf5NWq9sixagCZe5TzjdAdQjgPWXl63/W2SUSWJjzbMP5bYCGXqjqon'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 1, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 1, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '3cd904d7154e324a3d68e79f0d8e0c2fe65c4abb', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '3082387bba74fcd24c9700593d10418152d53c97', 'timestamp': '2021-03-09_20:35:33', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2137'} 2021-03-12T02:26:59.871 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-03-12T02:26:59.874 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-03-12T02:26:59.881 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/run/ceph 2021-03-12T02:26:59.923 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/run/ceph 2021-03-12T02:26:59.939 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2021-03-12T02:26:59.940 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2021-03-12T02:26:59.988 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-12T02:26:59.989 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-12T02:27:00.041 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-12T02:27:00.041 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-12T02:27:00.041 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 340 Links: 1 2021-03-12T02:27:00.042 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-12T02:27:00.042 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2021-03-12 02:24:00.841339962 +0000 2021-03-12T02:27:00.042 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2021-03-12 02:24:00.733339965 +0000 2021-03-12T02:27:00.042 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2021-03-12 02:24:00.733339965 +0000 2021-03-12T02:27:00.043 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2021-03-12T02:27:00.049 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2021-03-12T02:27:00.050 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2021-03-12T02:27:00.050 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.00037993 s, 1.3 MB/s 2021-03-12T02:27:00.054 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-12T02:27:00.101 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-12T02:27:00.101 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-12T02:27:00.101 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 349 Links: 1 2021-03-12T02:27:00.102 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-12T02:27:00.102 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2021-03-12 02:24:01.097339956 +0000 2021-03-12T02:27:00.102 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2021-03-12 02:24:01.001339959 +0000 2021-03-12T02:27:00.102 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2021-03-12 02:24:01.001339959 +0000 2021-03-12T02:27:00.103 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2021-03-12T02:27:00.111 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2021-03-12T02:27:00.111 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2021-03-12T02:27:00.111 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000273931 s, 1.9 MB/s 2021-03-12T02:27:00.114 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-12T02:27:00.161 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-12T02:27:00.162 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-12T02:27:00.162 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 355 Links: 1 2021-03-12T02:27:00.162 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-12T02:27:00.163 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2021-03-12 02:24:01.361339951 +0000 2021-03-12T02:27:00.163 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2021-03-12 02:24:01.245339953 +0000 2021-03-12T02:27:00.163 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2021-03-12 02:24:01.245339953 +0000 2021-03-12T02:27:00.163 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2021-03-12T02:27:00.173 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2021-03-12T02:27:00.173 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2021-03-12T02:27:00.174 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000355257 s, 1.4 MB/s 2021-03-12T02:27:00.176 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-12T02:27:00.225 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-12T02:27:00.225 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-12T02:27:00.226 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 361 Links: 1 2021-03-12T02:27:00.226 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-12T02:27:00.226 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2021-03-12 02:24:01.497339948 +0000 2021-03-12T02:27:00.226 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2021-03-12 02:24:01.493339948 +0000 2021-03-12T02:27:00.227 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2021-03-12 02:24:01.493339948 +0000 2021-03-12T02:27:00.227 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2021-03-12T02:27:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2021-03-12T02:27:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2021-03-12T02:27:00.235 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000392388 s, 1.3 MB/s 2021-03-12T02:27:00.239 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-12T02:27:00.240 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-03-12T02:27:00.240 INFO:tasks.ceph:Generating config... 2021-03-12T02:27:00.241 INFO:tasks.ceph:[client] debug rgw = 20 2021-03-12T02:27:00.242 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-03-12T02:27:00.242 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-03-12T02:27:00.242 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-12T02:27:00.243 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-03-12T02:27:00.243 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-03-12T02:27:00.243 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-03-12T02:27:00.244 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-03-12T02:27:00.244 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-03-12T02:27:00.244 INFO:tasks.ceph:[client] setgroup = ceph 2021-03-12T02:27:00.245 INFO:tasks.ceph:[client] setuser = ceph 2021-03-12T02:27:00.245 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2021-03-12T02:27:00.245 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-03-12T02:27:00.245 INFO:tasks.ceph:[mgr] debug ms = 1 2021-03-12T02:27:00.246 INFO:tasks.ceph:[mon] debug mon = 20 2021-03-12T02:27:00.246 INFO:tasks.ceph:[mon] debug ms = 1 2021-03-12T02:27:00.246 INFO:tasks.ceph:[mon] debug paxos = 20 2021-03-12T02:27:00.247 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2021-03-12T02:27:00.247 INFO:tasks.ceph:[osd] debug filestore = 20 2021-03-12T02:27:00.247 INFO:tasks.ceph:[osd] debug journal = 20 2021-03-12T02:27:00.248 INFO:tasks.ceph:[osd] debug ms = 1 2021-03-12T02:27:00.248 INFO:tasks.ceph:[osd] debug osd = 25 2021-03-12T02:27:00.248 INFO:tasks.ceph:[osd] osd fast shutdown = False 2021-03-12T02:27:00.249 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-03-12T02:27:00.249 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-03-12T02:27:00.250 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-03-12T02:27:00.250 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-03-12T02:27:00.250 INFO:tasks.ceph:Setting up c2.mon.a... 2021-03-12T02:27:00.251 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2021-03-12T02:27:00.314 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/c2.keyring 2021-03-12T02:27:00.316 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2021-03-12T02:27:00.393 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-03-12T02:27:00.449 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.114')] 2021-03-12T02:27:00.450 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.114 --print /home/ubuntu/cephtest/c2.monmap 2021-03-12T02:27:00.521 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2021-03-12T02:27:00.522 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool: generated fsid 3af5a41f-9507-40ef-950c-6ef173aee5a3 2021-03-12T02:27:00.522 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 0 2021-03-12T02:27:00.523 INFO:teuthology.orchestra.run.smithi114.stdout:fsid 3af5a41f-9507-40ef-950c-6ef173aee5a3 2021-03-12T02:27:00.523 INFO:teuthology.orchestra.run.smithi114.stdout:last_changed 2021-03-12T02:27:00.521220+0000 2021-03-12T02:27:00.524 INFO:teuthology.orchestra.run.smithi114.stdout:created 2021-03-12T02:27:00.521220+0000 2021-03-12T02:27:00.524 INFO:teuthology.orchestra.run.smithi114.stdout:min_mon_release 0 (unknown) 2021-03-12T02:27:00.524 INFO:teuthology.orchestra.run.smithi114.stdout:0: [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon.a 2021-03-12T02:27:00.525 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2021-03-12T02:27:00.525 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 3af5a41f-9507-40ef-950c-6ef173aee5a3... 2021-03-12T02:27:00.537 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-03-12T02:27:00.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-03-12T02:27:00.573 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2021-03-12T02:27:00.573 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2021-03-12T02:27:00.574 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-12T02:27:00.574 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2021-03-12T02:27:00.574 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.574 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2021-03-12T02:27:00.575 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.575 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2021-03-12T02:27:00.575 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.575 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2021-03-12T02:27:00.575 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2021-03-12T02:27:00.576 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.576 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2021-03-12T02:27:00.576 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2021-03-12T02:27:00.576 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.577 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-12T02:27:00.577 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2021-03-12T02:27:00.577 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.577 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2021-03-12T02:27:00.577 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-12T02:27:00.578 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.578 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2021-03-12T02:27:00.578 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.579 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2021-03-12T02:27:00.579 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2021-03-12T02:27:00.579 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2021-03-12T02:27:00.579 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2021-03-12T02:27:00.579 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2021-03-12T02:27:00.580 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2021-03-12T02:27:00.580 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2021-03-12T02:27:00.580 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-12T02:27:00.580 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-12T02:27:00.581 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.581 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-12T02:27:00.581 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.581 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2021-03-12T02:27:00.581 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.582 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2021-03-12T02:27:00.582 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.582 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2021-03-12T02:27:00.582 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2021-03-12T02:27:00.582 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2021-03-12T02:27:00.583 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = 172.21.15.114 2021-03-12T02:27:00.583 INFO:teuthology.orchestra.run.smithi077.stdout: osd heartbeat grace = 40 2021-03-12T02:27:00.583 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = 3af5a41f-9507-40ef-950c-6ef173aee5a3 2021-03-12T02:27:00.583 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.584 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2021-03-12T02:27:00.584 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2021-03-12T02:27:00.584 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.584 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2021-03-12T02:27:00.584 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2021-03-12T02:27:00.585 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.585 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2021-03-12T02:27:00.585 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2021-03-12T02:27:00.585 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.585 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2021-03-12T02:27:00.586 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2021-03-12T02:27:00.586 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2021-03-12T02:27:00.586 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.586 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2021-03-12T02:27:00.587 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2021-03-12T02:27:00.587 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.587 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2021-03-12T02:27:00.587 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.587 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2021-03-12T02:27:00.588 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.588 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2021-03-12T02:27:00.588 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.588 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2021-03-12T02:27:00.589 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2021-03-12T02:27:00.589 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.589 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2021-03-12T02:27:00.589 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2021-03-12T02:27:00.589 INFO:teuthology.orchestra.run.smithi077.stdout: debug filestore = 20 2021-03-12T02:27:00.590 INFO:teuthology.orchestra.run.smithi077.stdout: debug journal = 20 2021-03-12T02:27:00.590 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2021-03-12T02:27:00.590 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 25 2021-03-12T02:27:00.590 INFO:teuthology.orchestra.run.smithi077.stdout: osd fast shutdown = False 2021-03-12T02:27:00.590 INFO:teuthology.orchestra.run.smithi077.stdout: rocksdb delete range threshold = 0 2021-03-12T02:27:00.591 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.591 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2021-03-12T02:27:00.591 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2021-03-12T02:27:00.591 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2021-03-12T02:27:00.592 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2021-03-12T02:27:00.592 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2021-03-12T02:27:00.592 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2021-03-12T02:27:00.592 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2021-03-12T02:27:00.592 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2021-03-12T02:27:00.593 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.593 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2021-03-12T02:27:00.593 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2021-03-12T02:27:00.593 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2021-03-12T02:27:00.593 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2021-03-12T02:27:00.594 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2021-03-12T02:27:00.594 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2021-03-12T02:27:00.594 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2021-03-12T02:27:00.594 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2021-03-12T02:27:00.595 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2021-03-12T02:27:00.595 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2021-03-12T02:27:00.595 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2021-03-12T02:27:00.595 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd crush smoke test = False 2021-03-12T02:27:00.595 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-03-12T02:27:00.596 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2021-03-12T02:27:00.596 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2021-03-12T02:27:00.596 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2021-03-12T02:27:00.596 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2021-03-12T02:27:00.597 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-12T02:27:00.597 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-12T02:27:00.597 INFO:teuthology.orchestra.run.smithi077.stdout: debug rgw = 20 2021-03-12T02:27:00.597 INFO:teuthology.orchestra.run.smithi077.stdout: rgw crypt require ssl = False 2021-03-12T02:27:00.597 INFO:teuthology.orchestra.run.smithi077.stdout: rgw crypt s3 kms backend = testing 2021-03-12T02:27:00.598 INFO:teuthology.orchestra.run.smithi077.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-12T02:27:00.598 INFO:teuthology.orchestra.run.smithi077.stdout: rgw curl low speed time = 300 2021-03-12T02:27:00.598 INFO:teuthology.orchestra.run.smithi077.stdout: rgw data log num shards = 4 2021-03-12T02:27:00.598 INFO:teuthology.orchestra.run.smithi077.stdout: rgw md log max shards = 4 2021-03-12T02:27:00.599 INFO:teuthology.orchestra.run.smithi077.stdout: rgw sync log trim interval = 0 2021-03-12T02:27:00.599 INFO:teuthology.orchestra.run.smithi077.stdout: rgw sync obj etag verify = True 2021-03-12T02:27:00.599 INFO:teuthology.orchestra.run.smithi077.stdout: setgroup = ceph 2021-03-12T02:27:00.599 INFO:teuthology.orchestra.run.smithi077.stdout: setuser = ceph 2021-03-12T02:27:00.599 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2021-03-12T02:27:00.600 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.0] 2021-03-12T02:27:00.600 INFO:teuthology.orchestra.run.smithi077.stdout: osd_max_omap_entries_per_request = 10 2021-03-12T02:27:00.600 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.1] 2021-03-12T02:27:00.600 INFO:teuthology.orchestra.run.smithi077.stdout: osd_max_omap_entries_per_request = 1000 2021-03-12T02:27:00.600 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.2] 2021-03-12T02:27:00.601 INFO:teuthology.orchestra.run.smithi077.stdout: osd_max_omap_entries_per_request = 10000 2021-03-12T02:27:00.602 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2021-03-12T02:27:00.602 INFO:teuthology.orchestra.run.smithi114.stdout: chdir = "" 2021-03-12T02:27:00.602 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-12T02:27:00.602 INFO:teuthology.orchestra.run.smithi114.stdout: auth supported = cephx 2021-03-12T02:27:00.603 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.603 INFO:teuthology.orchestra.run.smithi114.stdout: filestore xattr use omap = true 2021-03-12T02:27:00.603 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.603 INFO:teuthology.orchestra.run.smithi114.stdout: mon clock drift allowed = 1.000 2021-03-12T02:27:00.604 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.604 INFO:teuthology.orchestra.run.smithi114.stdout: osd crush chooseleaf type = 0 2021-03-12T02:27:00.604 INFO:teuthology.orchestra.run.smithi114.stdout: auth debug = true 2021-03-12T02:27:00.604 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.604 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on old message = true 2021-03-12T02:27:00.605 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on bug = true 2021-03-12T02:27:00.605 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.605 INFO:teuthology.orchestra.run.smithi114.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-12T02:27:00.605 INFO:teuthology.orchestra.run.smithi114.stdout: mon pg warn max object skew = 0 2021-03-12T02:27:00.605 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.606 INFO:teuthology.orchestra.run.smithi114.stdout: # disable pg_autoscaler by default for new pools 2021-03-12T02:27:00.606 INFO:teuthology.orchestra.run.smithi114.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-12T02:27:00.606 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.606 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default size = 2 2021-03-12T02:27:00.607 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.607 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow primary affinity = true 2021-03-12T02:27:00.607 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow pg remap = true 2021-03-12T02:27:00.607 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on legacy crush tunables = false 2021-03-12T02:27:00.607 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on crush straw calc version zero = false 2021-03-12T02:27:00.608 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on no sortbitwise = false 2021-03-12T02:27:00.608 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on osd down out interval zero = false 2021-03-12T02:27:00.608 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on too few osds = false 2021-03-12T02:27:00.608 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-12T02:27:00.609 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-12T02:27:00.609 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.609 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-12T02:27:00.609 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.609 INFO:teuthology.orchestra.run.smithi114.stdout: osd default data pool replay window = 5 2021-03-12T02:27:00.610 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.610 INFO:teuthology.orchestra.run.smithi114.stdout: mon allow pool delete = true 2021-03-12T02:27:00.610 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.610 INFO:teuthology.orchestra.run.smithi114.stdout: mon cluster log file level = debug 2021-03-12T02:27:00.610 INFO:teuthology.orchestra.run.smithi114.stdout: debug asserts on shutdown = true 2021-03-12T02:27:00.611 INFO:teuthology.orchestra.run.smithi114.stdout: mon health detail to clog = false 2021-03-12T02:27:00.611 INFO:teuthology.orchestra.run.smithi114.stdout: mon host = 172.21.15.114 2021-03-12T02:27:00.611 INFO:teuthology.orchestra.run.smithi114.stdout: osd heartbeat grace = 40 2021-03-12T02:27:00.611 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = 3af5a41f-9507-40ef-950c-6ef173aee5a3 2021-03-12T02:27:00.612 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.612 INFO:teuthology.orchestra.run.smithi114.stdout:[osd] 2021-03-12T02:27:00.612 INFO:teuthology.orchestra.run.smithi114.stdout: osd journal size = 100 2021-03-12T02:27:00.612 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.612 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub load threshold = 5.0 2021-03-12T02:27:00.613 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub max interval = 600 2021-03-12T02:27:00.613 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.613 INFO:teuthology.orchestra.run.smithi114.stdout: osd recover clone overlap = true 2021-03-12T02:27:00.613 INFO:teuthology.orchestra.run.smithi114.stdout: osd recovery max chunk = 1048576 2021-03-12T02:27:00.613 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.614 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug shutdown = true 2021-03-12T02:27:00.614 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug op order = true 2021-03-12T02:27:00.614 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify stray on activate = true 2021-03-12T02:27:00.614 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.615 INFO:teuthology.orchestra.run.smithi114.stdout: osd open classes on start = true 2021-03-12T02:27:00.615 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug pg log writeout = true 2021-03-12T02:27:00.615 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.615 INFO:teuthology.orchestra.run.smithi114.stdout: osd deep scrub update digest min age = 30 2021-03-12T02:27:00.615 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.616 INFO:teuthology.orchestra.run.smithi114.stdout: osd map max advance = 10 2021-03-12T02:27:00.616 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.616 INFO:teuthology.orchestra.run.smithi114.stdout: journal zero on create = true 2021-03-12T02:27:00.616 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.616 INFO:teuthology.orchestra.run.smithi114.stdout: filestore ondisk finisher threads = 3 2021-03-12T02:27:00.617 INFO:teuthology.orchestra.run.smithi114.stdout: filestore apply finisher threads = 3 2021-03-12T02:27:00.617 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.617 INFO:teuthology.orchestra.run.smithi114.stdout: bdev debug aio = true 2021-03-12T02:27:00.617 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug misdirected ops = true 2021-03-12T02:27:00.618 INFO:teuthology.orchestra.run.smithi114.stdout: debug filestore = 20 2021-03-12T02:27:00.618 INFO:teuthology.orchestra.run.smithi114.stdout: debug journal = 20 2021-03-12T02:27:00.618 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2021-03-12T02:27:00.618 INFO:teuthology.orchestra.run.smithi114.stdout: debug osd = 25 2021-03-12T02:27:00.618 INFO:teuthology.orchestra.run.smithi114.stdout: osd fast shutdown = False 2021-03-12T02:27:00.619 INFO:teuthology.orchestra.run.smithi114.stdout: rocksdb delete range threshold = 0 2021-03-12T02:27:00.619 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.619 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr] 2021-03-12T02:27:00.619 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2021-03-12T02:27:00.619 INFO:teuthology.orchestra.run.smithi114.stdout: debug mgr = 20 2021-03-12T02:27:00.620 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2021-03-12T02:27:00.620 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2021-03-12T02:27:00.620 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2021-03-12T02:27:00.621 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2021-03-12T02:27:00.621 INFO:teuthology.orchestra.run.smithi114.stdout: mgr/telemetry/nag = false 2021-03-12T02:27:00.621 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.621 INFO:teuthology.orchestra.run.smithi114.stdout:[mon] 2021-03-12T02:27:00.622 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2021-03-12T02:27:00.622 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2021-03-12T02:27:00.622 INFO:teuthology.orchestra.run.smithi114.stdout: debug paxos = 20 2021-03-12T02:27:00.622 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2021-03-12T02:27:00.622 INFO:teuthology.orchestra.run.smithi114.stdout: mon data avail warn = 5 2021-03-12T02:27:00.623 INFO:teuthology.orchestra.run.smithi114.stdout: mon mgr mkfs grace = 240 2021-03-12T02:27:00.623 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2021-03-12T02:27:00.623 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd reporter subtree level = osd 2021-03-12T02:27:00.623 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd prime pg temp = true 2021-03-12T02:27:00.623 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2021-03-12T02:27:00.624 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd crush smoke test = False 2021-03-12T02:27:00.624 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:00.624 INFO:teuthology.orchestra.run.smithi114.stdout:[client] 2021-03-12T02:27:00.624 INFO:teuthology.orchestra.run.smithi114.stdout: rgw cache enabled = true 2021-03-12T02:27:00.625 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable ops log = true 2021-03-12T02:27:00.625 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable usage log = true 2021-03-12T02:27:00.625 INFO:teuthology.orchestra.run.smithi114.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-12T02:27:00.625 INFO:teuthology.orchestra.run.smithi114.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-12T02:27:00.625 INFO:teuthology.orchestra.run.smithi114.stdout: debug rgw = 20 2021-03-12T02:27:00.626 INFO:teuthology.orchestra.run.smithi114.stdout: rgw crypt require ssl = False 2021-03-12T02:27:00.626 INFO:teuthology.orchestra.run.smithi114.stdout: rgw crypt s3 kms backend = testing 2021-03-12T02:27:00.626 INFO:teuthology.orchestra.run.smithi114.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-12T02:27:00.626 INFO:teuthology.orchestra.run.smithi114.stdout: rgw curl low speed time = 300 2021-03-12T02:27:00.626 INFO:teuthology.orchestra.run.smithi114.stdout: rgw data log num shards = 4 2021-03-12T02:27:00.627 INFO:teuthology.orchestra.run.smithi114.stdout: rgw md log max shards = 4 2021-03-12T02:27:00.627 INFO:teuthology.orchestra.run.smithi114.stdout: rgw sync log trim interval = 0 2021-03-12T02:27:00.628 INFO:teuthology.orchestra.run.smithi114.stdout: rgw sync obj etag verify = True 2021-03-12T02:27:00.628 INFO:teuthology.orchestra.run.smithi114.stdout: setgroup = ceph 2021-03-12T02:27:00.628 INFO:teuthology.orchestra.run.smithi114.stdout: setuser = ceph 2021-03-12T02:27:00.628 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.a] 2021-03-12T02:27:00.628 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.0] 2021-03-12T02:27:00.629 INFO:teuthology.orchestra.run.smithi114.stdout: osd_max_omap_entries_per_request = 10 2021-03-12T02:27:00.629 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.1] 2021-03-12T02:27:00.629 INFO:teuthology.orchestra.run.smithi114.stdout: osd_max_omap_entries_per_request = 1000 2021-03-12T02:27:00.629 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.2] 2021-03-12T02:27:00.630 INFO:teuthology.orchestra.run.smithi114.stdout: osd_max_omap_entries_per_request = 10000 2021-03-12T02:27:00.631 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2021-03-12T02:27:00.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2021-03-12T02:27:00.720 INFO:tasks.ceph:Copying monmap to all nodes... 2021-03-12T02:27:00.731 DEBUG:teuthology.orchestra.remote:smithi114:/etc/ceph/c2.keyring is 206B 2021-03-12T02:27:00.791 DEBUG:teuthology.orchestra.remote:smithi114:/home/ubuntu/cephtest/c2.monmap is 179B 2021-03-12T02:27:00.846 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi077.front.sepia.ceph.com 2021-03-12T02:27:00.847 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2021-03-12T02:27:00.847 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/c2.keyring 2021-03-12T02:27:00.848 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-03-12T02:27:00.874 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2021-03-12T02:27:00.874 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-03-12T02:27:00.924 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi114.front.sepia.ceph.com 2021-03-12T02:27:00.924 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2021-03-12T02:27:00.925 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/c2.keyring 2021-03-12T02:27:00.926 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-03-12T02:27:00.950 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2021-03-12T02:27:00.951 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-03-12T02:27:00.997 INFO:tasks.ceph:Setting up mon nodes... 2021-03-12T02:27:00.997 INFO:tasks.ceph:Setting up mgr nodes... 2021-03-12T02:27:00.998 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2021-03-12T02:27:01.079 INFO:teuthology.orchestra.run.smithi114.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2021-03-12T02:27:01.081 INFO:tasks.ceph:Setting up mds nodes... 2021-03-12T02:27:01.081 INFO:tasks.ceph_client:Setting up client nodes... 2021-03-12T02:27:01.082 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2021-03-12T02:27:01.155 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/c2.client.0.keyring 2021-03-12T02:27:01.167 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2021-03-12T02:27:01.243 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/c2.client.1.keyring 2021-03-12T02:27:01.256 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-03-12T02:27:01.256 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-03-12T02:27:01.256 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2021-03-12T02:27:01.313 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-12T02:27:01.313 INFO:tasks.ceph:role: c2.osd.0 2021-03-12T02:27:01.314 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com 2021-03-12T02:27:01.314 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-03-12T02:27:01.834 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-03-12T02:27:01.834 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-12T02:27:01.834 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-12T02:27:01.835 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-12T02:27:01.835 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2021-03-12T02:27:01.835 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-12T02:27:01.835 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-12T02:27:01.835 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-12T02:27:01.836 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-12T02:27:01.837 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2021-03-12T02:27:01.837 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2021-03-12T02:27:01.922 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2021-03-12T02:27:01.934 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: /sbin/restorecon: command not found 2021-03-12T02:27:01.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-12T02:27:01.936 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2021-03-12T02:27:01.994 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-12T02:27:01.995 INFO:tasks.ceph:role: c2.osd.1 2021-03-12T02:27:01.995 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com 2021-03-12T02:27:01.995 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-03-12T02:27:02.515 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-03-12T02:27:02.515 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-12T02:27:02.516 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-12T02:27:02.516 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-12T02:27:02.516 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2021-03-12T02:27:02.516 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-12T02:27:02.517 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-12T02:27:02.517 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-12T02:27:02.517 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-12T02:27:02.518 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2021-03-12T02:27:02.518 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2021-03-12T02:27:02.572 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2021-03-12T02:27:02.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-12T02:27:02.586 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: /sbin/restorecon: command not found 2021-03-12T02:27:02.587 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2021-03-12T02:27:02.644 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-12T02:27:02.644 INFO:tasks.ceph:role: c2.osd.2 2021-03-12T02:27:02.644 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com 2021-03-12T02:27:02.645 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-03-12T02:27:03.143 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-03-12T02:27:03.143 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-12T02:27:03.144 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-12T02:27:03.144 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-12T02:27:03.145 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2021-03-12T02:27:03.145 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-12T02:27:03.146 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-12T02:27:03.146 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-12T02:27:03.146 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-12T02:27:03.148 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2021-03-12T02:27:03.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2021-03-12T02:27:03.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2021-03-12T02:27:03.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-12T02:27:03.219 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: /sbin/restorecon: command not found 2021-03-12T02:27:03.220 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2021-03-12T02:27:03.322 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:03.318+0000 7f8d25572d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2021-03-12T02:27:03.325 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:03.322+0000 7f8d25572d80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2021-03-12T02:27:03.325 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:03.322+0000 7f8d25572d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-03-12T02:27:03.325 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:03.322+0000 7f8d25572d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-03-12T02:27:03.325 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:03.322+0000 7f8d25572d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-03-12T02:27:03.327 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:03.326+0000 7f8d25572d80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2021-03-12T02:27:03.346 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:03.342+0000 7f8d25572d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-12T02:27:05.246 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2021-03-12T02:27:05.263 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2021-03-12T02:27:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:05.346+0000 7f69222b2d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2021-03-12T02:27:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:05.346+0000 7f69222b2d80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2021-03-12T02:27:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:05.346+0000 7f69222b2d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-03-12T02:27:05.349 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:05.346+0000 7f69222b2d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-03-12T02:27:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:05.346+0000 7f69222b2d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-03-12T02:27:05.350 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:05.346+0000 7f69222b2d80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2021-03-12T02:27:05.362 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:05.358+0000 7f69222b2d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-12T02:27:07.277 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2021-03-12T02:27:07.334 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2021-03-12T02:27:07.416 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:07.414+0000 7fa0a5b22d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2021-03-12T02:27:07.417 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:07.414+0000 7fa0a5b22d80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2021-03-12T02:27:07.417 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:07.414+0000 7fa0a5b22d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-03-12T02:27:07.417 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:07.414+0000 7fa0a5b22d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-03-12T02:27:07.417 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:07.414+0000 7fa0a5b22d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-03-12T02:27:07.418 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:07.414+0000 7fa0a5b22d80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2021-03-12T02:27:07.430 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:07.426+0000 7fa0a5b22d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-12T02:27:09.325 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2021-03-12T02:27:09.386 INFO:tasks.ceph:Reading keys from all nodes... 2021-03-12T02:27:09.386 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2021-03-12T02:27:09.432 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.PTLLuIx0ij 2021-03-12T02:27:09.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /var/lib/ceph/mgr/c2-x/keyring /tmp/tmp.PTLLuIx0ij 2021-03-12T02:27:09.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.PTLLuIx0ij 2021-03-12T02:27:09.602 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.PTLLuIx0ij is 56B 2021-03-12T02:27:09.613 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.PTLLuIx0ij 2021-03-12T02:27:09.619 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2021-03-12T02:27:09.671 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.GJdq5OXtef 2021-03-12T02:27:09.671 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /var/lib/ceph/osd/c2-0/keyring /tmp/tmp.GJdq5OXtef 2021-03-12T02:27:09.735 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.GJdq5OXtef 2021-03-12T02:27:09.843 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.GJdq5OXtef is 56B 2021-03-12T02:27:09.856 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.GJdq5OXtef 2021-03-12T02:27:09.865 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2021-03-12T02:27:09.912 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.xBbLqH5uEF 2021-03-12T02:27:09.913 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /var/lib/ceph/osd/c2-1/keyring /tmp/tmp.xBbLqH5uEF 2021-03-12T02:27:09.969 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.xBbLqH5uEF 2021-03-12T02:27:10.074 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.xBbLqH5uEF is 56B 2021-03-12T02:27:10.088 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.xBbLqH5uEF 2021-03-12T02:27:10.095 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2021-03-12T02:27:10.148 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.ZuAzNghZHj 2021-03-12T02:27:10.149 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /var/lib/ceph/osd/c2-2/keyring /tmp/tmp.ZuAzNghZHj 2021-03-12T02:27:10.205 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.ZuAzNghZHj 2021-03-12T02:27:10.318 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.ZuAzNghZHj is 56B 2021-03-12T02:27:10.339 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.ZuAzNghZHj 2021-03-12T02:27:10.396 DEBUG:teuthology.orchestra.remote:smithi114:/etc/ceph/c2.client.0.keyring is 59B 2021-03-12T02:27:10.418 DEBUG:teuthology.orchestra.remote:smithi114:/etc/ceph/c2.client.1.keyring is 59B 2021-03-12T02:27:10.471 INFO:tasks.ceph:Adding keys to all mons... 2021-03-12T02:27:10.472 DEBUG:teuthology.orchestra.run.smithi114:> sudo tee -a /etc/ceph/c2.keyring 2021-03-12T02:27:10.523 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr.x] 2021-03-12T02:27:10.523 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQB10UpgfPvbBBAAehgT/r4MRjiIH+vk6yZ1lA== 2021-03-12T02:27:10.523 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.0] 2021-03-12T02:27:10.524 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQB30UpghBNmExAA+oi0RWVTOnFqFSAL9nKtnA== 2021-03-12T02:27:10.524 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.1] 2021-03-12T02:27:10.524 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQB50UpgFgPwFBAAiY64LLnWn202sib5NHx53w== 2021-03-12T02:27:10.524 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.2] 2021-03-12T02:27:10.525 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQB70Upg4Db6GBAA7IaiJTDaIcIL51dmhKAnKw== 2021-03-12T02:27:10.525 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2021-03-12T02:27:10.525 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQB10Upgub9mCRAAR37SDSnnQ/siCWWADvSU6Q== 2021-03-12T02:27:10.525 INFO:teuthology.orchestra.run.smithi114.stdout:[client.1] 2021-03-12T02:27:10.525 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQB10UpgYaauDhAAtNQ3fs2PLC0HWt2jgvyQMA== 2021-03-12T02:27:10.527 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-12T02:27:10.612 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-12T02:27:10.696 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-12T02:27:10.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-12T02:27:10.870 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-12T02:27:10.959 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-12T02:27:11.002 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-03-12T02:27:11.003 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2021-03-12T02:27:11.061 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2021-03-12T02:27:11.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2021-03-12T02:27:11.195 DEBUG:teuthology.orchestra.run.smithi114:> rm -- /home/ubuntu/cephtest/c2.monmap 2021-03-12T02:27:11.251 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2021-03-12T02:27:11.251 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2021-03-12T02:27:11.252 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2021-03-12T02:27:11.296 INFO:tasks.ceph.c2.mon.a:Started 2021-03-12T02:27:11.297 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2021-03-12T02:27:11.297 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2021-03-12T02:27:11.298 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2021-03-12T02:27:11.300 INFO:tasks.ceph.c2.mgr.x:Started 2021-03-12T02:27:11.301 INFO:tasks.ceph:Setting crush tunables to default 2021-03-12T02:27:11.301 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster c2 osd crush tunables default 2021-03-12T02:27:11.595 INFO:teuthology.orchestra.run.smithi114.stderr:adjusted tunables profile to default 2021-03-12T02:27:11.606 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2021-03-12T02:27:11.607 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2021-03-12T02:27:11.613 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.gyZO5KrslC 2021-03-12T02:27:11.613 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /var/lib/ceph/osd/c2-0/fsid /tmp/tmp.gyZO5KrslC 2021-03-12T02:27:11.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.gyZO5KrslC 2021-03-12T02:27:11.778 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.gyZO5KrslC is 37B 2021-03-12T02:27:11.791 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.gyZO5KrslC 2021-03-12T02:27:11.799 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2021-03-12T02:27:11.848 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.puS3O4J65n 2021-03-12T02:27:11.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /var/lib/ceph/osd/c2-1/fsid /tmp/tmp.puS3O4J65n 2021-03-12T02:27:11.906 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.puS3O4J65n 2021-03-12T02:27:12.014 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.puS3O4J65n is 37B 2021-03-12T02:27:12.027 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.puS3O4J65n 2021-03-12T02:27:12.035 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2021-03-12T02:27:12.085 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.6qSXoC1ohK 2021-03-12T02:27:12.086 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /var/lib/ceph/osd/c2-2/fsid /tmp/tmp.6qSXoC1ohK 2021-03-12T02:27:12.142 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.6qSXoC1ohK 2021-03-12T02:27:12.246 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.6qSXoC1ohK is 37B 2021-03-12T02:27:12.257 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.6qSXoC1ohK 2021-03-12T02:27:12.263 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster c2 osd new a9592657-a41f-4357-b24b-10732fede8b5 0 2021-03-12T02:27:12.506 INFO:teuthology.orchestra.run.smithi114.stdout:0 2021-03-12T02:27:12.518 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster c2 osd new ca6c33f6-aee8-4ca1-89b1-132e0e0b13ec 1 2021-03-12T02:27:12.710 INFO:teuthology.orchestra.run.smithi114.stdout:1 2021-03-12T02:27:12.726 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster c2 osd new 64ae06d8-fa81-4876-97eb-3d79b544789b 2 2021-03-12T02:27:12.922 INFO:teuthology.orchestra.run.smithi114.stdout:2 2021-03-12T02:27:12.934 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2021-03-12T02:27:12.935 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2021-03-12T02:27:12.937 INFO:tasks.ceph.c2.osd.0:Started 2021-03-12T02:27:12.937 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2021-03-12T02:27:12.938 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2021-03-12T02:27:12.940 INFO:tasks.ceph.c2.osd.1:Started 2021-03-12T02:27:12.941 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2021-03-12T02:27:12.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2021-03-12T02:27:12.943 INFO:tasks.ceph.c2.osd.2:Started 2021-03-12T02:27:12.944 INFO:tasks.ceph:Waiting for OSDs to come up 2021-03-12T02:27:13.247 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-03-12T02:27:13.320 INFO:tasks.ceph.c2.osd.0.smithi114.stderr:2021-03-12T02:27:13.318+0000 7fc74a735d80 -1 Falling back to public interface 2021-03-12T02:27:13.414 INFO:teuthology.misc.health.smithi114.stdout: 2021-03-12T02:27:13.415 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":5,"fsid":"3af5a41f-9507-40ef-950c-6ef173aee5a3","created":"2021-03-12T02:27:11.428478+0000","modified":"2021-03-12T02:27:12.922919+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"a9592657-a41f-4357-b24b-10732fede8b5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ca6c33f6-aee8-4ca1-89b1-132e0e0b13ec","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"64ae06d8-fa81-4876-97eb-3d79b544789b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-12T02:27:13.425 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-03-12T02:27:13.537 INFO:tasks.ceph.c2.osd.1.smithi114.stderr:2021-03-12T02:27:13.534+0000 7ff7ef647d80 -1 Falling back to public interface 2021-03-12T02:27:13.572 INFO:tasks.ceph.c2.osd.2.smithi114.stderr:2021-03-12T02:27:13.570+0000 7fdc68cebd80 -1 Falling back to public interface 2021-03-12T02:27:13.944 INFO:tasks.ceph.c2.osd.0.smithi114.stderr:2021-03-12T02:27:13.942+0000 7fc74a735d80 -1 osd.0 0 log_to_monitors {default=true} 2021-03-12T02:27:14.156 INFO:tasks.ceph.c2.osd.1.smithi114.stderr:2021-03-12T02:27:14.154+0000 7ff7ef647d80 -1 osd.1 0 log_to_monitors {default=true} 2021-03-12T02:27:14.204 INFO:tasks.ceph.c2.osd.2.smithi114.stderr:2021-03-12T02:27:14.202+0000 7fdc68cebd80 -1 osd.2 0 log_to_monitors {default=true} 2021-03-12T02:27:15.468 INFO:tasks.ceph.c2.osd.2.smithi114.stderr:2021-03-12T02:27:15.466+0000 7fdc52737700 -1 osd.2 0 waiting for initial osdmap 2021-03-12T02:27:15.469 INFO:tasks.ceph.c2.osd.0.smithi114.stderr:2021-03-12T02:27:15.466+0000 7fc734181700 -1 osd.0 0 waiting for initial osdmap 2021-03-12T02:27:15.469 INFO:tasks.ceph.c2.osd.1.smithi114.stderr:2021-03-12T02:27:15.466+0000 7ff7d9093700 -1 osd.1 0 waiting for initial osdmap 2021-03-12T02:27:19.728 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-03-12T02:27:19.995 INFO:teuthology.misc.health.smithi114.stdout: 2021-03-12T02:27:19.995 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"3af5a41f-9507-40ef-950c-6ef173aee5a3","created":"2021-03-12T02:27:11.428478+0000","modified":"2021-03-12T02:27:18.485370+0000","last_up_change":"2021-03-12T02:27:16.465903+0000","last_in_change":"2021-03-12T02:27:16.465903+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-12T02:27:15.178499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"a9592657-a41f-4357-b24b-10732fede8b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":13576},{"type":"v1","addr":"172.21.15.114:6801","nonce":13576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":13576},{"type":"v1","addr":"172.21.15.114:6803","nonce":13576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":13576},{"type":"v1","addr":"172.21.15.114:6807","nonce":13576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":13576},{"type":"v1","addr":"172.21.15.114:6805","nonce":13576}]},"public_addr":"172.21.15.114:6801/13576","cluster_addr":"172.21.15.114:6803/13576","heartbeat_back_addr":"172.21.15.114:6807/13576","heartbeat_front_addr":"172.21.15.114:6805/13576","state":["exists","up"]},{"osd":1,"uuid":"ca6c33f6-aee8-4ca1-89b1-132e0e0b13ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":13575},{"type":"v1","addr":"172.21.15.114:6809","nonce":13575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":13575},{"type":"v1","addr":"172.21.15.114:6811","nonce":13575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":13575},{"type":"v1","addr":"172.21.15.114:6815","nonce":13575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":13575},{"type":"v1","addr":"172.21.15.114:6813","nonce":13575}]},"public_addr":"172.21.15.114:6809/13575","cluster_addr":"172.21.15.114:6811/13575","heartbeat_back_addr":"172.21.15.114:6815/13575","heartbeat_front_addr":"172.21.15.114:6813/13575","state":["exists","up"]},{"osd":2,"uuid":"64ae06d8-fa81-4876-97eb-3d79b544789b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":13599},{"type":"v1","addr":"172.21.15.114:6817","nonce":13599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":13599},{"type":"v1","addr":"172.21.15.114:6819","nonce":13599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":13599},{"type":"v1","addr":"172.21.15.114:6823","nonce":13599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":13599},{"type":"v1","addr":"172.21.15.114:6821","nonce":13599}]},"public_addr":"172.21.15.114:6817/13599","cluster_addr":"172.21.15.114:6819/13599","heartbeat_back_addr":"172.21.15.114:6823/13599","heartbeat_front_addr":"172.21.15.114:6821/13599","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:27:14.987133+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:27:15.184632+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:27:15.223047+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-12T02:27:20.008 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-03-12T02:27:20.009 INFO:tasks.ceph:Creating RBD pool 2021-03-12T02:27:20.009 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster c2 osd pool create rbd 8 2021-03-12T02:27:20.511 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'rbd' created 2021-03-12T02:27:21.457 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-03-12T02:27:22.519 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rbd' on pool 'rbd' 2021-03-12T02:27:22.533 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2021-03-12T02:27:22.535 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2021-03-12T02:27:22.536 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-03-12T02:27:22.804 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:22.804 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":13,"fsid":"3af5a41f-9507-40ef-950c-6ef173aee5a3","created":"2021-03-12T02:27:11.428478+0000","modified":"2021-03-12T02:27:22.510337+0000","last_up_change":"2021-03-12T02:27:16.465903+0000","last_in_change":"2021-03-12T02:27:16.465903+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-12T02:27:15.178499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-12T02:27:20.273855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a9592657-a41f-4357-b24b-10732fede8b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":13576},{"type":"v1","addr":"172.21.15.114:6801","nonce":13576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":13576},{"type":"v1","addr":"172.21.15.114:6803","nonce":13576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":13576},{"type":"v1","addr":"172.21.15.114:6807","nonce":13576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":13576},{"type":"v1","addr":"172.21.15.114:6805","nonce":13576}]},"public_addr":"172.21.15.114:6801/13576","cluster_addr":"172.21.15.114:6803/13576","heartbeat_back_addr":"172.21.15.114:6807/13576","heartbeat_front_addr":"172.21.15.114:6805/13576","state":["exists","up"]},{"osd":1,"uuid":"ca6c33f6-aee8-4ca1-89b1-132e0e0b13ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":13575},{"type":"v1","addr":"172.21.15.114:6809","nonce":13575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":13575},{"type":"v1","addr":"172.21.15.114:6811","nonce":13575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":13575},{"type":"v1","addr":"172.21.15.114:6815","nonce":13575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":13575},{"type":"v1","addr":"172.21.15.114:6813","nonce":13575}]},"public_addr":"172.21.15.114:6809/13575","cluster_addr":"172.21.15.114:6811/13575","heartbeat_back_addr":"172.21.15.114:6815/13575","heartbeat_front_addr":"172.21.15.114:6813/13575","state":["exists","up"]},{"osd":2,"uuid":"64ae06d8-fa81-4876-97eb-3d79b544789b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":13599},{"type":"v1","addr":"172.21.15.114:6817","nonce":13599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":13599},{"type":"v1","addr":"172.21.15.114:6819","nonce":13599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":13599},{"type":"v1","addr":"172.21.15.114:6823","nonce":13599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":13599},{"type":"v1","addr":"172.21.15.114:6821","nonce":13599}]},"public_addr":"172.21.15.114:6817/13599","cluster_addr":"172.21.15.114:6819/13599","heartbeat_back_addr":"172.21.15.114:6823/13599","heartbeat_front_addr":"172.21.15.114:6821/13599","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:27:14.987133+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:27:15.184632+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:27:15.223047+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-12T02:27:22.817 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-12T02:27:15.178499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-03-12T02:27:20.273855+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '13', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-03-12T02:27:22.818 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get device_health_metrics pg_num 2021-03-12T02:27:23.092 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 1 2021-03-12T02:27:23.104 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get rbd pg_num 2021-03-12T02:27:23.379 INFO:teuthology.orchestra.run.smithi114.stdout:pg_num: 8 2021-03-12T02:27:23.392 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2021-03-12T02:27:23.393 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2021-03-12T02:27:23.394 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early mgr dump --format=json 2021-03-12T02:27:23.692 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:23.705 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6824","nonce":13366},{"type":"v1","addr":"172.21.15.114:6825","nonce":13366}]},"active_addr":"172.21.15.114:6825/13366","active_change":"2021-03-12T02:27:15.147580+0000","active_mgr_features":4540138292840890367,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":4126574953}]},{"addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":278953527}]},{"addrvec":[{"type":"v2","addr":"172.21.15.114:0","nonce":114000611}]}]}} 2021-03-12T02:27:23.707 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2021-03-12T02:27:23.708 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2021-03-12T02:27:23.708 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2021-03-12T02:27:23.975 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:23.975 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":13,"fsid":"3af5a41f-9507-40ef-950c-6ef173aee5a3","created":"2021-03-12T02:27:11.428478+0000","modified":"2021-03-12T02:27:22.510337+0000","last_up_change":"2021-03-12T02:27:16.465903+0000","last_in_change":"2021-03-12T02:27:16.465903+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-12T02:27:15.178499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-12T02:27:20.273855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a9592657-a41f-4357-b24b-10732fede8b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":13576},{"type":"v1","addr":"172.21.15.114:6801","nonce":13576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":13576},{"type":"v1","addr":"172.21.15.114:6803","nonce":13576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":13576},{"type":"v1","addr":"172.21.15.114:6807","nonce":13576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":13576},{"type":"v1","addr":"172.21.15.114:6805","nonce":13576}]},"public_addr":"172.21.15.114:6801/13576","cluster_addr":"172.21.15.114:6803/13576","heartbeat_back_addr":"172.21.15.114:6807/13576","heartbeat_front_addr":"172.21.15.114:6805/13576","state":["exists","up"]},{"osd":1,"uuid":"ca6c33f6-aee8-4ca1-89b1-132e0e0b13ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":13575},{"type":"v1","addr":"172.21.15.114:6809","nonce":13575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":13575},{"type":"v1","addr":"172.21.15.114:6811","nonce":13575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":13575},{"type":"v1","addr":"172.21.15.114:6815","nonce":13575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":13575},{"type":"v1","addr":"172.21.15.114:6813","nonce":13575}]},"public_addr":"172.21.15.114:6809/13575","cluster_addr":"172.21.15.114:6811/13575","heartbeat_back_addr":"172.21.15.114:6815/13575","heartbeat_front_addr":"172.21.15.114:6813/13575","state":["exists","up"]},{"osd":2,"uuid":"64ae06d8-fa81-4876-97eb-3d79b544789b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":13599},{"type":"v1","addr":"172.21.15.114:6817","nonce":13599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":13599},{"type":"v1","addr":"172.21.15.114:6819","nonce":13599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":13599},{"type":"v1","addr":"172.21.15.114:6823","nonce":13599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":13599},{"type":"v1","addr":"172.21.15.114:6821","nonce":13599}]},"public_addr":"172.21.15.114:6817/13599","cluster_addr":"172.21.15.114:6819/13599","heartbeat_back_addr":"172.21.15.114:6823/13599","heartbeat_front_addr":"172.21.15.114:6821/13599","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:27:14.987133+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:27:15.184632+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:27:15.223047+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-12T02:27:23.988 INFO:tasks.ceph.ceph_manager.c2:all up! 2021-03-12T02:27:23.989 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2021-03-12T02:27:24.256 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:24.256 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":13,"fsid":"3af5a41f-9507-40ef-950c-6ef173aee5a3","created":"2021-03-12T02:27:11.428478+0000","modified":"2021-03-12T02:27:22.510337+0000","last_up_change":"2021-03-12T02:27:16.465903+0000","last_in_change":"2021-03-12T02:27:16.465903+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-12T02:27:15.178499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-12T02:27:20.273855+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a9592657-a41f-4357-b24b-10732fede8b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":13576},{"type":"v1","addr":"172.21.15.114:6801","nonce":13576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":13576},{"type":"v1","addr":"172.21.15.114:6803","nonce":13576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":13576},{"type":"v1","addr":"172.21.15.114:6807","nonce":13576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":13576},{"type":"v1","addr":"172.21.15.114:6805","nonce":13576}]},"public_addr":"172.21.15.114:6801/13576","cluster_addr":"172.21.15.114:6803/13576","heartbeat_back_addr":"172.21.15.114:6807/13576","heartbeat_front_addr":"172.21.15.114:6805/13576","state":["exists","up"]},{"osd":1,"uuid":"ca6c33f6-aee8-4ca1-89b1-132e0e0b13ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":13575},{"type":"v1","addr":"172.21.15.114:6809","nonce":13575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":13575},{"type":"v1","addr":"172.21.15.114:6811","nonce":13575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":13575},{"type":"v1","addr":"172.21.15.114:6815","nonce":13575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":13575},{"type":"v1","addr":"172.21.15.114:6813","nonce":13575}]},"public_addr":"172.21.15.114:6809/13575","cluster_addr":"172.21.15.114:6811/13575","heartbeat_back_addr":"172.21.15.114:6815/13575","heartbeat_front_addr":"172.21.15.114:6813/13575","state":["exists","up"]},{"osd":2,"uuid":"64ae06d8-fa81-4876-97eb-3d79b544789b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":13599},{"type":"v1","addr":"172.21.15.114:6817","nonce":13599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":13599},{"type":"v1","addr":"172.21.15.114:6819","nonce":13599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":13599},{"type":"v1","addr":"172.21.15.114:6823","nonce":13599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":13599},{"type":"v1","addr":"172.21.15.114:6821","nonce":13599}]},"public_addr":"172.21.15.114:6817/13599","cluster_addr":"172.21.15.114:6819/13599","heartbeat_back_addr":"172.21.15.114:6823/13599","heartbeat_front_addr":"172.21.15.114:6821/13599","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:27:14.987133+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:27:15.184632+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-12T02:27:15.223047+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-12T02:27:24.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.0 flush_pg_stats 2021-03-12T02:27:24.428 INFO:teuthology.orchestra.run.smithi114.stdout:34359738372 2021-03-12T02:27:24.428 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.1 flush_pg_stats 2021-03-12T02:27:24.583 INFO:teuthology.orchestra.run.smithi114.stdout:34359738371 2021-03-12T02:27:24.583 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.2 flush_pg_stats 2021-03-12T02:27:24.744 INFO:teuthology.orchestra.run.smithi114.stdout:34359738372 2021-03-12T02:27:24.744 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2021-03-12T02:27:25.013 INFO:teuthology.orchestra.run.smithi114.stdout:34359738371 2021-03-12T02:27:25.026 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738371 for osd.0 2021-03-12T02:27:26.028 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2021-03-12T02:27:26.320 INFO:teuthology.orchestra.run.smithi114.stdout:34359738372 2021-03-12T02:27:26.334 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.0 2021-03-12T02:27:26.335 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.1 2021-03-12T02:27:26.607 INFO:teuthology.orchestra.run.smithi114.stdout:34359738371 2021-03-12T02:27:26.620 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.1 2021-03-12T02:27:26.621 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.2 2021-03-12T02:27:26.886 INFO:teuthology.orchestra.run.smithi114.stdout:34359738372 2021-03-12T02:27:26.899 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.2 2021-03-12T02:27:26.900 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2021-03-12T02:27:26.901 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2021-03-12T02:27:27.167 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:27.168 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2021-03-12T02:27:27.176 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-03-12T02:27:27.158503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146080,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426720,"statfs":{"total":322122547200,"available":318900961280,"internally_reserved":3221225472,"allocated":335872,"data_stored":94848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.645473"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.535885+0000","last_change":"2021-03-12T02:27:21.530733+0000","last_active":"2021-03-12T02:27:21.535885+0000","last_peered":"2021-03-12T02:27:21.535885+0000","last_clean":"2021-03-12T02:27:21.535885+0000","last_became_active":"2021-03-12T02:27:21.530427+0000","last_became_peered":"2021-03-12T02:27:21.530427+0000","last_unstale":"2021-03-12T02:27:21.535885+0000","last_undegraded":"2021-03-12T02:27:21.535885+0000","last_fullsized":"2021-03-12T02:27:21.535885+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_clean_scrub_stamp":"2021-03-12T02:27:20.496769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.535515+0000","last_change":"2021-03-12T02:27:21.528838+0000","last_active":"2021-03-12T02:27:21.535515+0000","last_peered":"2021-03-12T02:27:21.535515+0000","last_clean":"2021-03-12T02:27:21.535515+0000","last_became_active":"2021-03-12T02:27:21.528233+0000","last_became_peered":"2021-03-12T02:27:21.528233+0000","last_unstale":"2021-03-12T02:27:21.535515+0000","last_undegraded":"2021-03-12T02:27:21.535515+0000","last_fullsized":"2021-03-12T02:27:21.535515+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_clean_scrub_stamp":"2021-03-12T02:27:20.496769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.535507+0000","last_change":"2021-03-12T02:27:21.528646+0000","last_active":"2021-03-12T02:27:21.535507+0000","last_peered":"2021-03-12T02:27:21.535507+0000","last_clean":"2021-03-12T02:27:21.535507+0000","last_became_active":"2021-03-12T02:27:21.528228+0000","last_became_peered":"2021-03-12T02:27:21.528228+0000","last_unstale":"2021-03-12T02:27:21.535507+0000","last_undegraded":"2021-03-12T02:27:21.535507+0000","last_fullsized":"2021-03-12T02:27:21.535507+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_clean_scrub_stamp":"2021-03-12T02:27:20.496769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.527521+0000","last_change":"2021-03-12T02:27:21.524871+0000","last_active":"2021-03-12T02:27:21.527521+0000","last_peered":"2021-03-12T02:27:21.527521+0000","last_clean":"2021-03-12T02:27:21.527521+0000","last_became_active":"2021-03-12T02:27:21.524361+0000","last_became_peered":"2021-03-12T02:27:21.524361+0000","last_unstale":"2021-03-12T02:27:21.527521+0000","last_undegraded":"2021-03-12T02:27:21.527521+0000","last_fullsized":"2021-03-12T02:27:21.527521+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_clean_scrub_stamp":"2021-03-12T02:27:20.496769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.523689+0000","last_change":"2021-03-12T02:27:17.569624+0000","last_active":"2021-03-12T02:27:21.523689+0000","last_peered":"2021-03-12T02:27:21.523689+0000","last_clean":"2021-03-12T02:27:21.523689+0000","last_became_active":"2021-03-12T02:27:17.568378+0000","last_became_peered":"2021-03-12T02:27:17.568378+0000","last_unstale":"2021-03-12T02:27:21.523689+0000","last_undegraded":"2021-03-12T02:27:21.523689+0000","last_fullsized":"2021-03-12T02:27:21.523689+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:15.454653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:15.454653+0000","last_clean_scrub_stamp":"2021-03-12T02:27:15.454653+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.530436+0000","last_change":"2021-03-12T02:27:21.524653+0000","last_active":"2021-03-12T02:27:21.530436+0000","last_peered":"2021-03-12T02:27:21.530436+0000","last_clean":"2021-03-12T02:27:21.530436+0000","last_became_active":"2021-03-12T02:27:21.524072+0000","last_became_peered":"2021-03-12T02:27:21.524072+0000","last_unstale":"2021-03-12T02:27:21.530436+0000","last_undegraded":"2021-03-12T02:27:21.530436+0000","last_fullsized":"2021-03-12T02:27:21.530436+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_clean_scrub_stamp":"2021-03-12T02:27:20.496769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.530141+0000","last_change":"2021-03-12T02:27:21.526449+0000","last_active":"2021-03-12T02:27:21.530141+0000","last_peered":"2021-03-12T02:27:21.530141+0000","last_clean":"2021-03-12T02:27:21.530141+0000","last_became_active":"2021-03-12T02:27:21.526043+0000","last_became_peered":"2021-03-12T02:27:21.526043+0000","last_unstale":"2021-03-12T02:27:21.530141+0000","last_undegraded":"2021-03-12T02:27:21.530141+0000","last_fullsized":"2021-03-12T02:27:21.530141+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_clean_scrub_stamp":"2021-03-12T02:27:20.496769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.528348+0000","last_change":"2021-03-12T02:27:21.528348+0000","last_active":"2021-03-12T02:27:21.528348+0000","last_peered":"2021-03-12T02:27:21.528348+0000","last_clean":"2021-03-12T02:27:21.528348+0000","last_became_active":"2021-03-12T02:27:21.528006+0000","last_became_peered":"2021-03-12T02:27:21.528006+0000","last_unstale":"2021-03-12T02:27:21.528348+0000","last_undegraded":"2021-03-12T02:27:21.528348+0000","last_fullsized":"2021-03-12T02:27:21.528348+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_clean_scrub_stamp":"2021-03-12T02:27:20.496769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.535893+0000","last_change":"2021-03-12T02:27:21.530827+0000","last_active":"2021-03-12T02:27:21.535893+0000","last_peered":"2021-03-12T02:27:21.535893+0000","last_clean":"2021-03-12T02:27:21.535893+0000","last_became_active":"2021-03-12T02:27:21.530324+0000","last_became_peered":"2021-03-12T02:27:21.530324+0000","last_unstale":"2021-03-12T02:27:21.535893+0000","last_undegraded":"2021-03-12T02:27:21.535893+0000","last_fullsized":"2021-03-12T02:27:21.535893+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_clean_scrub_stamp":"2021-03-12T02:27:20.496769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":32787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":32787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]}]}]}} 2021-03-12T02:27:27.178 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2021-03-12T02:27:27.445 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:27.447 INFO:teuthology.orchestra.run.smithi114.stderr:dumped all 2021-03-12T02:27:27.466 INFO:teuthology.orchestra.run.smithi114.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-03-12T02:27:27.158503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146080,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426720,"statfs":{"total":322122547200,"available":318900961280,"internally_reserved":3221225472,"allocated":335872,"data_stored":94848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.645473"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.535885+0000","last_change":"2021-03-12T02:27:21.530733+0000","last_active":"2021-03-12T02:27:21.535885+0000","last_peered":"2021-03-12T02:27:21.535885+0000","last_clean":"2021-03-12T02:27:21.535885+0000","last_became_active":"2021-03-12T02:27:21.530427+0000","last_became_peered":"2021-03-12T02:27:21.530427+0000","last_unstale":"2021-03-12T02:27:21.535885+0000","last_undegraded":"2021-03-12T02:27:21.535885+0000","last_fullsized":"2021-03-12T02:27:21.535885+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_clean_scrub_stamp":"2021-03-12T02:27:20.496769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.535515+0000","last_change":"2021-03-12T02:27:21.528838+0000","last_active":"2021-03-12T02:27:21.535515+0000","last_peered":"2021-03-12T02:27:21.535515+0000","last_clean":"2021-03-12T02:27:21.535515+0000","last_became_active":"2021-03-12T02:27:21.528233+0000","last_became_peered":"2021-03-12T02:27:21.528233+0000","last_unstale":"2021-03-12T02:27:21.535515+0000","last_undegraded":"2021-03-12T02:27:21.535515+0000","last_fullsized":"2021-03-12T02:27:21.535515+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_clean_scrub_stamp":"2021-03-12T02:27:20.496769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.535507+0000","last_change":"2021-03-12T02:27:21.528646+0000","last_active":"2021-03-12T02:27:21.535507+0000","last_peered":"2021-03-12T02:27:21.535507+0000","last_clean":"2021-03-12T02:27:21.535507+0000","last_became_active":"2021-03-12T02:27:21.528228+0000","last_became_peered":"2021-03-12T02:27:21.528228+0000","last_unstale":"2021-03-12T02:27:21.535507+0000","last_undegraded":"2021-03-12T02:27:21.535507+0000","last_fullsized":"2021-03-12T02:27:21.535507+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_clean_scrub_stamp":"2021-03-12T02:27:20.496769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.527521+0000","last_change":"2021-03-12T02:27:21.524871+0000","last_active":"2021-03-12T02:27:21.527521+0000","last_peered":"2021-03-12T02:27:21.527521+0000","last_clean":"2021-03-12T02:27:21.527521+0000","last_became_active":"2021-03-12T02:27:21.524361+0000","last_became_peered":"2021-03-12T02:27:21.524361+0000","last_unstale":"2021-03-12T02:27:21.527521+0000","last_undegraded":"2021-03-12T02:27:21.527521+0000","last_fullsized":"2021-03-12T02:27:21.527521+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_clean_scrub_stamp":"2021-03-12T02:27:20.496769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.523689+0000","last_change":"2021-03-12T02:27:17.569624+0000","last_active":"2021-03-12T02:27:21.523689+0000","last_peered":"2021-03-12T02:27:21.523689+0000","last_clean":"2021-03-12T02:27:21.523689+0000","last_became_active":"2021-03-12T02:27:17.568378+0000","last_became_peered":"2021-03-12T02:27:17.568378+0000","last_unstale":"2021-03-12T02:27:21.523689+0000","last_undegraded":"2021-03-12T02:27:21.523689+0000","last_fullsized":"2021-03-12T02:27:21.523689+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:15.454653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:15.454653+0000","last_clean_scrub_stamp":"2021-03-12T02:27:15.454653+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.530436+0000","last_change":"2021-03-12T02:27:21.524653+0000","last_active":"2021-03-12T02:27:21.530436+0000","last_peered":"2021-03-12T02:27:21.530436+0000","last_clean":"2021-03-12T02:27:21.530436+0000","last_became_active":"2021-03-12T02:27:21.524072+0000","last_became_peered":"2021-03-12T02:27:21.524072+0000","last_unstale":"2021-03-12T02:27:21.530436+0000","last_undegraded":"2021-03-12T02:27:21.530436+0000","last_fullsized":"2021-03-12T02:27:21.530436+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_clean_scrub_stamp":"2021-03-12T02:27:20.496769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.530141+0000","last_change":"2021-03-12T02:27:21.526449+0000","last_active":"2021-03-12T02:27:21.530141+0000","last_peered":"2021-03-12T02:27:21.530141+0000","last_clean":"2021-03-12T02:27:21.530141+0000","last_became_active":"2021-03-12T02:27:21.526043+0000","last_became_peered":"2021-03-12T02:27:21.526043+0000","last_unstale":"2021-03-12T02:27:21.530141+0000","last_undegraded":"2021-03-12T02:27:21.530141+0000","last_fullsized":"2021-03-12T02:27:21.530141+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_clean_scrub_stamp":"2021-03-12T02:27:20.496769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.528348+0000","last_change":"2021-03-12T02:27:21.528348+0000","last_active":"2021-03-12T02:27:21.528348+0000","last_peered":"2021-03-12T02:27:21.528348+0000","last_clean":"2021-03-12T02:27:21.528348+0000","last_became_active":"2021-03-12T02:27:21.528006+0000","last_became_peered":"2021-03-12T02:27:21.528006+0000","last_unstale":"2021-03-12T02:27:21.528348+0000","last_undegraded":"2021-03-12T02:27:21.528348+0000","last_fullsized":"2021-03-12T02:27:21.528348+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_clean_scrub_stamp":"2021-03-12T02:27:20.496769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-12T02:27:21.535893+0000","last_change":"2021-03-12T02:27:21.530827+0000","last_active":"2021-03-12T02:27:21.535893+0000","last_peered":"2021-03-12T02:27:21.535893+0000","last_clean":"2021-03-12T02:27:21.535893+0000","last_became_active":"2021-03-12T02:27:21.530324+0000","last_became_peered":"2021-03-12T02:27:21.530324+0000","last_unstale":"2021-03-12T02:27:21.535893+0000","last_undegraded":"2021-03-12T02:27:21.535893+0000","last_fullsized":"2021-03-12T02:27:21.535893+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-12T02:27:20.496769+0000","last_clean_scrub_stamp":"2021-03-12T02:27:20.496769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":32787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":32787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]}]}]}} 2021-03-12T02:27:27.467 INFO:tasks.ceph.ceph_manager.c2:clean! 2021-03-12T02:27:27.467 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2021-03-12T02:27:27.468 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2021-03-12T02:27:27.468 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early health --format=json 2021-03-12T02:27:27.784 INFO:teuthology.orchestra.run.smithi114.stdout: 2021-03-12T02:27:27.798 INFO:teuthology.orchestra.run.smithi114.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-03-12T02:27:27.798 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2021-03-12T02:27:27.799 INFO:teuthology.run_tasks:Running task rgw... 2021-03-12T02:27:27.813 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2021-03-12T02:27:27.814 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2021-03-12T02:27:27.814 INFO:tasks.rgw:Creating data pools 2021-03-12T02:27:27.815 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2021-03-12T02:27:27.815 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-03-12T02:27:28.185 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'default.rgw.buckets.data' created 2021-03-12T02:27:28.201 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-03-12T02:27:29.202 INFO:teuthology.orchestra.run.smithi077.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-03-12T02:27:29.221 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-03-12T02:27:30.211 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'default.rgw.buckets.index' created 2021-03-12T02:27:30.224 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-03-12T02:27:31.231 INFO:teuthology.orchestra.run.smithi077.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-03-12T02:27:31.243 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2021-03-12T02:27:31.244 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-03-12T02:27:31.514 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'default.rgw.buckets.data' already exists 2021-03-12T02:27:31.524 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-03-12T02:27:32.221 INFO:teuthology.orchestra.run.smithi077.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-03-12T02:27:32.233 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-03-12T02:27:32.509 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'default.rgw.buckets.index' already exists 2021-03-12T02:27:32.522 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-03-12T02:27:33.221 INFO:teuthology.orchestra.run.smithi077.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-03-12T02:27:33.233 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2021-03-12T02:27:33.234 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-03-12T02:27:34.415 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'default.rgw.buckets.data' created 2021-03-12T02:27:34.429 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-03-12T02:27:35.453 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-03-12T02:27:35.464 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-03-12T02:27:36.439 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'default.rgw.buckets.index' created 2021-03-12T02:27:36.453 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-03-12T02:27:37.472 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-03-12T02:27:37.478 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2021-03-12T02:27:37.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-03-12T02:27:37.744 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'default.rgw.buckets.data' already exists 2021-03-12T02:27:37.759 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-03-12T02:27:38.436 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-03-12T02:27:38.449 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-03-12T02:27:38.717 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'default.rgw.buckets.index' already exists 2021-03-12T02:27:38.728 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-03-12T02:27:39.455 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-03-12T02:27:39.468 DEBUG:tasks.rgw:Pools created 2021-03-12T02:27:39.468 INFO:tasks.rgw:Configuring compression type = random 2021-03-12T02:27:39.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2021-03-12T02:27:39.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2021-03-12T02:27:39.470 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2021-03-12T02:27:39.512 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:27:39.512 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:27:39.523 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:39.521+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:41.168 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.169+0000 7fa4ba3ab080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:41.169 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.169+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:41.169 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.173+0000 7fa4ba3ab080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:41.170 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.173+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:41.170 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.173+0000 7fa4ba3ab080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:41.170 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.173+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:41.171 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.173+0000 7fa4ba3ab080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:41.184 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.185+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:41.185 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.185+0000 7fa4ba3ab080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:41.185 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.185+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:41.185 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.185+0000 7fa4ba3ab080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:41.186 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.185+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:41.186 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.189+0000 7fa4ba3ab080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:41.186 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.189+0000 7fa4ba3ab080 10 cannot find current period zonegroup using local zonegroup 2021-03-12T02:27:41.186 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.189+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:41.187 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.189+0000 7fa4ba3ab080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:41.187 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.189+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:41.188 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.189+0000 7fa4ba3ab080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:41.188 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.189+0000 7fa4ba3ab080 10 Creating default zonegroup 2021-03-12T02:27:41.188 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.189+0000 7fa4ba3ab080 20 get_system_obj_state: rctx=0x7ffd3036ea60 obj=:.pools.avail state=0x55e835a02ec0 s->prefetch_data=0 2021-03-12T02:27:41.189 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.189+0000 7fa4ba3ab080 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-03-12T02:27:41.196 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.197+0000 7fa4ba3ab080 10 WARNING: store->list_zones() returned r=-2 2021-03-12T02:27:41.197 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.197+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:41.197 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.201+0000 7fa4ba3ab080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:41.200 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.201+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:41.201 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.201+0000 7fa4ba3ab080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:41.201 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.201+0000 7fa4ba3ab080 10 could not read realm id: (2) No such file or directory 2021-03-12T02:27:41.201 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.201+0000 7fa4ba3ab080 10 WARNING: failed to set zone as default, r=-22 2021-03-12T02:27:41.201 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.201+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:41.202 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.205+0000 7fa4ba3ab080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:41.204 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.205+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:41.205 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.209+0000 7fa4ba3ab080 20 rados_obj.operate() r=0 bl.length=839 2021-03-12T02:27:41.205 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.209+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.209+0000 7fa4ba3ab080 20 rados_obj.operate() r=0 bl.length=358 2021-03-12T02:27:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.209+0000 7fa4ba3ab080 20 zonegroup default 2021-03-12T02:27:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.209+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:41.207 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.209+0000 7fa4ba3ab080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:41.207 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.209+0000 7fa4ba3ab080 10 Cannot find current period zone using local zone 2021-03-12T02:27:41.207 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.209+0000 7fa4ba3ab080 10 Using default name default 2021-03-12T02:27:41.207 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.209+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.209+0000 7fa4ba3ab080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:27:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.209+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.209+0000 7fa4ba3ab080 20 rados_obj.operate() r=0 bl.length=839 2021-03-12T02:27:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.209+0000 7fa4ba3ab080 20 zone default found 2021-03-12T02:27:41.210 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:41.209+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:43.183 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:43.185+0000 7fa4ba3ab080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:43.183 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:43.185+0000 7fa4ba3ab080 20 rados->read ofs=0 len=0 2021-03-12T02:27:43.184 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:43.185+0000 7fa4ba3ab080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:43.184 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:43.185+0000 7fa4ba3ab080 20 started sync module instance, tier type = 2021-03-12T02:27:43.184 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:43.185+0000 7fa4ba3ab080 20 started zone id=97fe90bf-8b97-42a2-ad49-6c621dc1fa1f (name=default) with tier type = 2021-03-12T02:27:45.234 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:45.237+0000 7fa4ba3ab080 20 add_watcher() i=0 2021-03-12T02:27:45.234 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:45.237+0000 7fa4ba3ab080 20 add_watcher() i=1 2021-03-12T02:27:45.235 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:45.237+0000 7fa4ba3ab080 20 add_watcher() i=2 2021-03-12T02:27:45.235 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:45.237+0000 7fa4ba3ab080 20 add_watcher() i=3 2021-03-12T02:27:45.235 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:45.237+0000 7fa4ba3ab080 20 add_watcher() i=4 2021-03-12T02:27:45.235 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:45.237+0000 7fa4ba3ab080 20 add_watcher() i=5 2021-03-12T02:27:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:45.237+0000 7fa4ba3ab080 20 add_watcher() i=6 2021-03-12T02:27:45.236 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:45.237+0000 7fa4ba3ab080 20 add_watcher() i=7 2021-03-12T02:27:45.237 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:45.237+0000 7fa4ba3ab080 2 all 8 watchers are set, enabling cache 2021-03-12T02:27:45.237 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:45.241+0000 7fa4ba3ab080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-12T02:27:45.237 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:45.241+0000 7fa4ba3ab080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-12T02:27:45.237 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:45.241+0000 7fa4767ec700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-12T02:27:49.220 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.221+0000 7fa4ba3ab080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-03-12T02:27:49.413 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.417+0000 7fa3ecff9700 20 reqs_thread_entry: start 2021-03-12T02:27:49.414 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.417+0000 7fa4ba3ab080 20 init_complete bucket index max shards: 11 2021-03-12T02:27:49.414 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.417+0000 7fa3da7fc700 20 reqs_thread_entry: start 2021-03-12T02:27:49.414 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.417+0000 7fa3d8ff9700 20 reqs_thread_entry: start 2021-03-12T02:27:49.435 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.437+0000 7fa4767ec700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-12T02:27:49.437 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.441+0000 7fa4ba3ab080 20 remove_watcher() i=0 2021-03-12T02:27:49.437 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.441+0000 7fa4ba3ab080 2 removed watcher, disabling cache 2021-03-12T02:27:49.439 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.441+0000 7fa4ba3ab080 20 remove_watcher() i=1 2021-03-12T02:27:49.442 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.445+0000 7fa4ba3ab080 20 remove_watcher() i=2 2021-03-12T02:27:49.445 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.445+0000 7fa4ba3ab080 20 remove_watcher() i=3 2021-03-12T02:27:49.446 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.449+0000 7fa4ba3ab080 20 remove_watcher() i=4 2021-03-12T02:27:49.450 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.453+0000 7fa4ba3ab080 20 remove_watcher() i=5 2021-03-12T02:27:49.451 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.453+0000 7fa4ba3ab080 20 remove_watcher() i=6 2021-03-12T02:27:49.454 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.457+0000 7fa4ba3ab080 20 remove_watcher() i=7 2021-03-12T02:27:49.462 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2021-03-12T02:27:49.462 DEBUG:tasks.util.rgw: json result: [] 2021-03-12T02:27:49.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-12T02:27:49.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-12T02:27:49.464 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-03-12T02:27:49.543 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:27:49.543 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:27:49.556 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.557+0000 7f2505808080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-12T02:27:49.556 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.557+0000 7f2505808080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-12T02:27:49.565 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.565+0000 7f2439ffb700 20 reqs_thread_entry: start 2021-03-12T02:27:49.568 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.565+0000 7f2505808080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.573 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.565+0000 7f2505808080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:27:49.575 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.565+0000 7f2505808080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.579 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.569+0000 7f2505808080 20 rados_obj.operate() r=0 bl.length=839 2021-03-12T02:27:49.581 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.569+0000 7f2505808080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.581 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.569+0000 7f2505808080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:49.581 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.569+0000 7f2505808080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.582 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.569+0000 7f2505808080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:27:49.582 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.569+0000 7f2505808080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.582 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.569+0000 7f2505808080 20 rados_obj.operate() r=0 bl.length=358 2021-03-12T02:27:49.584 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"97fe90bf-8b97-42a2-ad49-6c621dc1fa1f","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-12T02:27:49.584 DEBUG:tasks.util.rgw: json result: {'id': '97fe90bf-8b97-42a2-ad49-6c621dc1fa1f', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2021-03-12T02:27:49.585 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2021-03-12T02:27:49.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2021-03-12T02:27:49.586 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2021-03-12T02:27:49.658 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:27:49.658 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:27:49.671 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.673+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.675 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.677+0000 7f74f1294080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:49.675 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.677+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.675 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.677+0000 7f74f1294080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:49.675 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.677+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.676 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.677+0000 7f74f1294080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:49.676 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.677+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.676 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.677+0000 7f74f1294080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:27:49.694 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.697+0000 7f74f1294080 20 RGWRados::pool_iterate: got zonegroup_info.97723bd4-7f4a-4317-8f3b-7e9e21f5ab8b 2021-03-12T02:27:49.695 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.697+0000 7f74f1294080 20 RGWRados::pool_iterate: got zone_info.97fe90bf-8b97-42a2-ad49-6c621dc1fa1f 2021-03-12T02:27:49.695 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.697+0000 7f74f1294080 20 RGWRados::pool_iterate: got zone_names.default 2021-03-12T02:27:49.695 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.697+0000 7f74f1294080 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-03-12T02:27:49.696 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.697+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.696 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.697+0000 7f74f1294080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:27:49.696 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.697+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.698 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.701+0000 7f74f1294080 20 rados_obj.operate() r=0 bl.length=855 2021-03-12T02:27:49.699 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.701+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.700 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.701+0000 7f74f1294080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:27:49.701 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.701+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.701 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.701+0000 7f74f1294080 20 rados_obj.operate() r=0 bl.length=358 2021-03-12T02:27:49.701 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.701+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.702 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.701+0000 7f74f1294080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:49.702 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.701+0000 7f74f1294080 10 cannot find current period zonegroup using local zonegroup 2021-03-12T02:27:49.702 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.701+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.706 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.705+0000 7f74f1294080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:49.706 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.705+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.706 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.705+0000 7f74f1294080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:27:49.706 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.705+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.707 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.705+0000 7f74f1294080 20 rados_obj.operate() r=0 bl.length=358 2021-03-12T02:27:49.707 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.705+0000 7f74f1294080 20 zonegroup default 2021-03-12T02:27:49.707 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.705+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.707 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.705+0000 7f74f1294080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:49.707 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.705+0000 7f74f1294080 10 Cannot find current period zone using local zone 2021-03-12T02:27:49.708 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.705+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.708 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.705+0000 7f74f1294080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:49.708 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.705+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.708 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.705+0000 7f74f1294080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:27:49.709 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.705+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.709 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.705+0000 7f74f1294080 20 rados_obj.operate() r=0 bl.length=855 2021-03-12T02:27:49.709 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.705+0000 7f74f1294080 20 zone default found 2021-03-12T02:27:49.711 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.705+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.711 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.709+0000 7f74f1294080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:49.712 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.709+0000 7f74f1294080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.712 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.709+0000 7f74f1294080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:49.712 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.709+0000 7f74f1294080 20 started sync module instance, tier type = 2021-03-12T02:27:49.713 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.709+0000 7f74f1294080 20 started zone id=97fe90bf-8b97-42a2-ad49-6c621dc1fa1f (name=default) with tier type = 2021-03-12T02:27:49.736 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.737+0000 7f74f1294080 20 add_watcher() i=0 2021-03-12T02:27:49.737 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.737+0000 7f74f1294080 20 add_watcher() i=1 2021-03-12T02:27:49.737 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.737+0000 7f74f1294080 20 add_watcher() i=2 2021-03-12T02:27:49.737 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.737+0000 7f74f1294080 20 add_watcher() i=3 2021-03-12T02:27:49.737 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.737+0000 7f74f1294080 20 add_watcher() i=4 2021-03-12T02:27:49.738 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.737+0000 7f74f1294080 20 add_watcher() i=5 2021-03-12T02:27:49.738 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.737+0000 7f74f1294080 20 add_watcher() i=6 2021-03-12T02:27:49.738 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.737+0000 7f74f1294080 20 add_watcher() i=7 2021-03-12T02:27:49.738 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.737+0000 7f74f1294080 2 all 8 watchers are set, enabling cache 2021-03-12T02:27:49.739 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.741+0000 7f74f1294080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-12T02:27:49.739 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.741+0000 7f74f1294080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-12T02:27:49.739 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.741+0000 7f74ad7ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-12T02:27:49.824 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.825+0000 7f741f7fe700 20 reqs_thread_entry: start 2021-03-12T02:27:49.824 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.825+0000 7f741d7fa700 20 reqs_thread_entry: start 2021-03-12T02:27:49.824 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.825+0000 7f74f1294080 20 init_complete bucket index max shards: 11 2021-03-12T02:27:49.825 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.825+0000 7f740b7fe700 20 reqs_thread_entry: start 2021-03-12T02:27:49.845 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.849+0000 7f74ad7ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-12T02:27:49.846 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.849+0000 7f74f1294080 20 remove_watcher() i=0 2021-03-12T02:27:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.849+0000 7f74f1294080 2 removed watcher, disabling cache 2021-03-12T02:27:49.849 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.853+0000 7f74f1294080 20 remove_watcher() i=1 2021-03-12T02:27:49.852 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.853+0000 7f74f1294080 20 remove_watcher() i=2 2021-03-12T02:27:49.854 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.857+0000 7f74f1294080 20 remove_watcher() i=3 2021-03-12T02:27:49.856 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.861+0000 7f74f1294080 20 remove_watcher() i=4 2021-03-12T02:27:49.859 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.861+0000 7f74f1294080 20 remove_watcher() i=5 2021-03-12T02:27:49.860 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.865+0000 7f74f1294080 20 remove_watcher() i=6 2021-03-12T02:27:49.862 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.865+0000 7f74f1294080 20 remove_watcher() i=7 2021-03-12T02:27:49.870 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2021-03-12T02:27:49.871 DEBUG:tasks.util.rgw: json result: [] 2021-03-12T02:27:49.871 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-12T02:27:49.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-12T02:27:49.872 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-03-12T02:27:49.953 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:27:49.953 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:27:49.966 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.969+0000 7f5cc8c76080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-12T02:27:49.966 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.969+0000 7f5cc8c76080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-12T02:27:49.972 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.973+0000 7f5cc8c76080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.973 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.973+0000 7f5c257fa700 20 reqs_thread_entry: start 2021-03-12T02:27:49.973 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.977+0000 7f5cc8c76080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:27:49.974 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.977+0000 7f5cc8c76080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.975 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.977+0000 7f5cc8c76080 20 rados_obj.operate() r=0 bl.length=855 2021-03-12T02:27:49.976 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.977+0000 7f5cc8c76080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.976 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.977+0000 7f5cc8c76080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:49.976 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.977+0000 7f5cc8c76080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.976 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.977+0000 7f5cc8c76080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:27:49.977 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.977+0000 7f5cc8c76080 20 rados->read ofs=0 len=0 2021-03-12T02:27:49.978 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:27:49.977+0000 7f5cc8c76080 20 rados_obj.operate() r=0 bl.length=358 2021-03-12T02:27:49.986 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"97fe90bf-8b97-42a2-ad49-6c621dc1fa1f","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-12T02:27:49.986 DEBUG:tasks.util.rgw: json result: {'id': '97fe90bf-8b97-42a2-ad49-6c621dc1fa1f', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2021-03-12T02:27:49.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2021-03-12T02:27:49.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2021-03-12T02:27:49.988 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2021-03-12T02:27:50.023 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:27:50.024 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:27:50.035 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:50.034+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:51.495 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.494+0000 7f53f8dda080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.494+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:51.734 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.494+0000 7f53f8dda080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:51.735 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.494+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:51.739 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.494+0000 7f53f8dda080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:51.739 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.494+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:51.739 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.494+0000 7f53f8dda080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:51.740 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.506+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:51.740 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.510+0000 7f53f8dda080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:51.740 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.510+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:51.740 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.510+0000 7f53f8dda080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:51.740 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.510+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:51.741 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.510+0000 7f53f8dda080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:51.741 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.510+0000 7f53f8dda080 10 cannot find current period zonegroup using local zonegroup 2021-03-12T02:27:51.741 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.510+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:51.741 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.510+0000 7f53f8dda080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:51.742 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.510+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:51.742 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.510+0000 7f53f8dda080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:51.742 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.510+0000 7f53f8dda080 10 Creating default zonegroup 2021-03-12T02:27:51.742 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.510+0000 7f53f8dda080 20 get_system_obj_state: rctx=0x7ffcc201d550 obj=:.pools.avail state=0x5581c4d02ec0 s->prefetch_data=0 2021-03-12T02:27:51.743 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.510+0000 7f53f8dda080 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-03-12T02:27:51.743 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.522+0000 7f53f8dda080 10 WARNING: store->list_zones() returned r=-2 2021-03-12T02:27:51.743 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.522+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:51.743 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.522+0000 7f53f8dda080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:51.744 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.526+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:51.744 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.526+0000 7f53f8dda080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:51.744 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.526+0000 7f53f8dda080 10 could not read realm id: (2) No such file or directory 2021-03-12T02:27:51.744 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.526+0000 7f53f8dda080 10 WARNING: failed to set zone as default, r=-22 2021-03-12T02:27:51.744 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.526+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:51.745 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.526+0000 7f53f8dda080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:51.745 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.530+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:51.745 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.534+0000 7f53f8dda080 20 rados_obj.operate() r=0 bl.length=839 2021-03-12T02:27:51.745 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.534+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:51.748 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.534+0000 7f53f8dda080 20 rados_obj.operate() r=0 bl.length=358 2021-03-12T02:27:51.748 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.534+0000 7f53f8dda080 20 zonegroup default 2021-03-12T02:27:51.748 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.534+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:51.749 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.534+0000 7f53f8dda080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:51.749 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.534+0000 7f53f8dda080 10 Cannot find current period zone using local zone 2021-03-12T02:27:51.749 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.534+0000 7f53f8dda080 10 Using default name default 2021-03-12T02:27:51.749 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.534+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:51.750 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.534+0000 7f53f8dda080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:27:51.750 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.534+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:51.750 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.534+0000 7f53f8dda080 20 rados_obj.operate() r=0 bl.length=839 2021-03-12T02:27:51.751 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.534+0000 7f53f8dda080 20 zone default found 2021-03-12T02:27:51.751 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:51.534+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:53.510 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:53.510+0000 7f53f8dda080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:53.511 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:53.510+0000 7f53f8dda080 20 rados->read ofs=0 len=0 2021-03-12T02:27:53.511 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:53.510+0000 7f53f8dda080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:27:53.512 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:53.510+0000 7f53f8dda080 20 started sync module instance, tier type = 2021-03-12T02:27:53.512 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:53.510+0000 7f53f8dda080 20 started zone id=ba43ae6d-2028-42dd-ae44-7f517e1f167f (name=default) with tier type = 2021-03-12T02:27:55.559 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:55.558+0000 7f53f8dda080 20 add_watcher() i=0 2021-03-12T02:27:55.560 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:55.558+0000 7f53f8dda080 20 add_watcher() i=1 2021-03-12T02:27:55.560 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:55.558+0000 7f53f8dda080 20 add_watcher() i=2 2021-03-12T02:27:55.560 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:55.558+0000 7f53f8dda080 20 add_watcher() i=3 2021-03-12T02:27:55.560 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:55.558+0000 7f53f8dda080 20 add_watcher() i=4 2021-03-12T02:27:55.561 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:55.558+0000 7f53f8dda080 20 add_watcher() i=5 2021-03-12T02:27:55.561 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:55.558+0000 7f53f8dda080 20 add_watcher() i=6 2021-03-12T02:27:55.562 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:55.558+0000 7f53f8dda080 20 add_watcher() i=7 2021-03-12T02:27:55.562 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:55.558+0000 7f53f8dda080 2 all 8 watchers are set, enabling cache 2021-03-12T02:27:55.562 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:55.558+0000 7f53f8dda080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-12T02:27:55.562 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:55.558+0000 7f53f8dda080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-12T02:27:55.563 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:55.562+0000 7f53b5fdb700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-12T02:27:59.548 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:59.546+0000 7f53f8dda080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-03-12T02:28:00.517 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:59.646+0000 7f531bfff700 20 reqs_thread_entry: start 2021-03-12T02:28:00.517 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:59.646+0000 7f5319ffb700 20 reqs_thread_entry: start 2021-03-12T02:28:00.517 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:59.646+0000 7f53f8dda080 20 init_complete bucket index max shards: 11 2021-03-12T02:28:00.517 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:59.646+0000 7f530bfff700 20 reqs_thread_entry: start 2021-03-12T02:28:00.519 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:59.666+0000 7f53b5fdb700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-12T02:28:00.520 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:59.670+0000 7f53f8dda080 20 remove_watcher() i=0 2021-03-12T02:28:00.520 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:59.670+0000 7f53f8dda080 2 removed watcher, disabling cache 2021-03-12T02:28:00.520 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:59.670+0000 7f53f8dda080 20 remove_watcher() i=1 2021-03-12T02:28:00.520 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:59.674+0000 7f53f8dda080 20 remove_watcher() i=2 2021-03-12T02:28:00.521 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:59.678+0000 7f53f8dda080 20 remove_watcher() i=3 2021-03-12T02:28:00.522 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:59.682+0000 7f53f8dda080 20 remove_watcher() i=4 2021-03-12T02:28:00.522 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:59.686+0000 7f53f8dda080 20 remove_watcher() i=5 2021-03-12T02:28:00.523 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:59.690+0000 7f53f8dda080 20 remove_watcher() i=6 2021-03-12T02:28:00.523 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:27:59.690+0000 7f53f8dda080 20 remove_watcher() i=7 2021-03-12T02:28:00.523 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2021-03-12T02:28:00.524 DEBUG:tasks.util.rgw: json result: [] 2021-03-12T02:28:00.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-12T02:28:00.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-12T02:28:00.525 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-03-12T02:28:00.600 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:00.601 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:00.611 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.610+0000 7f41ded98080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-12T02:28:00.611 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.610+0000 7f41ded98080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-12T02:28:00.619 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.618+0000 7f41137fe700 20 reqs_thread_entry: start 2021-03-12T02:28:00.619 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.618+0000 7f41ded98080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.620 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.618+0000 7f41ded98080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:28:00.620 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.618+0000 7f41ded98080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.621 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.618+0000 7f41ded98080 20 rados_obj.operate() r=0 bl.length=839 2021-03-12T02:28:00.621 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.618+0000 7f41ded98080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.622 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.618+0000 7f41ded98080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:28:00.622 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.618+0000 7f41ded98080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.623 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.622+0000 7f41ded98080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:28:00.623 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.622+0000 7f41ded98080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.623 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.622+0000 7f41ded98080 20 rados_obj.operate() r=0 bl.length=358 2021-03-12T02:28:00.632 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"ba43ae6d-2028-42dd-ae44-7f517e1f167f","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-12T02:28:00.632 DEBUG:tasks.util.rgw: json result: {'id': 'ba43ae6d-2028-42dd-ae44-7f517e1f167f', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2021-03-12T02:28:00.633 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2021-03-12T02:28:00.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2021-03-12T02:28:00.633 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2021-03-12T02:28:00.704 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:00.704 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:00.713 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.710+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.714 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.714+0000 7fecce80d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:28:00.714 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.714+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.716 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.714+0000 7fecce80d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:28:00.716 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.714+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.716 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.714+0000 7fecce80d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:28:00.717 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.714+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.717 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.714+0000 7fecce80d080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:28:00.731 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.730+0000 7fecce80d080 20 RGWRados::pool_iterate: got zone_info.ba43ae6d-2028-42dd-ae44-7f517e1f167f 2021-03-12T02:28:00.731 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.730+0000 7fecce80d080 20 RGWRados::pool_iterate: got zonegroup_info.89c11b75-2a9b-4d6d-8f89-6ca03704b11e 2021-03-12T02:28:00.732 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.730+0000 7fecce80d080 20 RGWRados::pool_iterate: got zone_names.default 2021-03-12T02:28:00.732 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.730+0000 7fecce80d080 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-03-12T02:28:00.732 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.730+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.733 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.730+0000 7fecce80d080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:28:00.734 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.730+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.734 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.730+0000 7fecce80d080 20 rados_obj.operate() r=0 bl.length=855 2021-03-12T02:28:00.734 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.730+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.734 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.730+0000 7fecce80d080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:28:00.735 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.730+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.735 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.730+0000 7fecce80d080 20 rados_obj.operate() r=0 bl.length=358 2021-03-12T02:28:00.735 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.730+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.735 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.730+0000 7fecce80d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:28:00.735 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.730+0000 7fecce80d080 10 cannot find current period zonegroup using local zonegroup 2021-03-12T02:28:00.736 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.730+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.737 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:28:00.737 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.738 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:28:00.738 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.738 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 rados_obj.operate() r=0 bl.length=358 2021-03-12T02:28:00.738 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 zonegroup default 2021-03-12T02:28:00.739 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.739 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:28:00.739 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 10 Cannot find current period zone using local zone 2021-03-12T02:28:00.739 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.740 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:28:00.740 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.740 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:28:00.740 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.741 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 rados_obj.operate() r=0 bl.length=855 2021-03-12T02:28:00.741 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 zone default found 2021-03-12T02:28:00.741 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.741 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:28:00.742 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 rados->read ofs=0 len=0 2021-03-12T02:28:00.742 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:28:00.742 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 started sync module instance, tier type = 2021-03-12T02:28:00.742 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.734+0000 7fecce80d080 20 started zone id=ba43ae6d-2028-42dd-ae44-7f517e1f167f (name=default) with tier type = 2021-03-12T02:28:00.768 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.766+0000 7fecce80d080 20 add_watcher() i=0 2021-03-12T02:28:00.768 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.766+0000 7fecce80d080 20 add_watcher() i=1 2021-03-12T02:28:00.768 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.766+0000 7fecce80d080 20 add_watcher() i=2 2021-03-12T02:28:00.768 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.766+0000 7fecce80d080 20 add_watcher() i=3 2021-03-12T02:28:00.769 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.766+0000 7fecce80d080 20 add_watcher() i=4 2021-03-12T02:28:00.769 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.766+0000 7fecce80d080 20 add_watcher() i=5 2021-03-12T02:28:00.769 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.766+0000 7fecce80d080 20 add_watcher() i=6 2021-03-12T02:28:00.770 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.766+0000 7fecce80d080 20 add_watcher() i=7 2021-03-12T02:28:00.770 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.766+0000 7fecce80d080 2 all 8 watchers are set, enabling cache 2021-03-12T02:28:00.771 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.770+0000 7fecce80d080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-12T02:28:00.771 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.770+0000 7fecce80d080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-12T02:28:00.771 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.770+0000 7fec8a7ec700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-12T02:28:00.870 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.866+0000 7fec00ff9700 20 reqs_thread_entry: start 2021-03-12T02:28:00.870 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.866+0000 7febfa7fc700 20 reqs_thread_entry: start 2021-03-12T02:28:00.870 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.866+0000 7fecce80d080 20 init_complete bucket index max shards: 11 2021-03-12T02:28:00.871 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.870+0000 7febf8ff9700 20 reqs_thread_entry: start 2021-03-12T02:28:00.892 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.890+0000 7fec8a7ec700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-12T02:28:00.894 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.894+0000 7fecce80d080 20 remove_watcher() i=0 2021-03-12T02:28:00.894 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.894+0000 7fecce80d080 2 removed watcher, disabling cache 2021-03-12T02:28:00.896 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.894+0000 7fecce80d080 20 remove_watcher() i=1 2021-03-12T02:28:00.899 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.898+0000 7fecce80d080 20 remove_watcher() i=2 2021-03-12T02:28:00.901 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.898+0000 7fecce80d080 20 remove_watcher() i=3 2021-03-12T02:28:00.903 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.902+0000 7fecce80d080 20 remove_watcher() i=4 2021-03-12T02:28:00.906 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.902+0000 7fecce80d080 20 remove_watcher() i=5 2021-03-12T02:28:00.908 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.906+0000 7fecce80d080 20 remove_watcher() i=6 2021-03-12T02:28:00.910 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:00.906+0000 7fecce80d080 20 remove_watcher() i=7 2021-03-12T02:28:00.920 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2021-03-12T02:28:00.921 DEBUG:tasks.util.rgw: json result: [] 2021-03-12T02:28:00.921 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-12T02:28:00.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-12T02:28:00.922 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-03-12T02:28:01.002 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:01.002 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:01.010+0000 7fd2c35f6080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-12T02:28:01.013 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:01.010+0000 7fd2c35f6080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-12T02:28:01.019 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:01.018+0000 7fd2c35f6080 20 rados->read ofs=0 len=0 2021-03-12T02:28:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:01.018+0000 7fd1f6ffd700 20 reqs_thread_entry: start 2021-03-12T02:28:01.020 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:01.018+0000 7fd2c35f6080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:28:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:01.018+0000 7fd2c35f6080 20 rados->read ofs=0 len=0 2021-03-12T02:28:01.021 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:01.018+0000 7fd2c35f6080 20 rados_obj.operate() r=0 bl.length=855 2021-03-12T02:28:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:01.018+0000 7fd2c35f6080 20 rados->read ofs=0 len=0 2021-03-12T02:28:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:01.018+0000 7fd2c35f6080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-12T02:28:01.022 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:01.018+0000 7fd2c35f6080 20 rados->read ofs=0 len=0 2021-03-12T02:28:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:01.022+0000 7fd2c35f6080 20 rados_obj.operate() r=0 bl.length=46 2021-03-12T02:28:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:01.022+0000 7fd2c35f6080 20 rados->read ofs=0 len=0 2021-03-12T02:28:01.023 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:01.022+0000 7fd2c35f6080 20 rados_obj.operate() r=0 bl.length=358 2021-03-12T02:28:01.032 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"ba43ae6d-2028-42dd-ae44-7f517e1f167f","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-12T02:28:01.033 DEBUG:tasks.util.rgw: json result: {'id': 'ba43ae6d-2028-42dd-ae44-7f517e1f167f', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2021-03-12T02:28:01.033 INFO:tasks.rgw:Starting rgw... 2021-03-12T02:28:01.033 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2021-03-12T02:28:01.034 INFO:tasks.rgw:Using civetweb as radosgw frontend 2021-03-12T02:28:01.034 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-03-12T02:28:01.035 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-03-12T02:28:01.039 INFO:tasks.rgw.c1.client.0:Started 2021-03-12T02:28:01.039 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2021-03-12T02:28:01.039 INFO:tasks.rgw:Using civetweb as radosgw frontend 2021-03-12T02:28:01.040 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-03-12T02:28:01.040 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-03-12T02:28:01.046 INFO:tasks.rgw.c1.client.1:Started 2021-03-12T02:28:01.050 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2021-03-12T02:28:01.050 INFO:tasks.rgw:Using civetweb as radosgw frontend 2021-03-12T02:28:01.052 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-03-12T02:28:01.052 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-03-12T02:28:01.076 INFO:tasks.rgw.c2.client.0:Started 2021-03-12T02:28:01.076 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2021-03-12T02:28:01.076 INFO:tasks.rgw:Using civetweb as radosgw frontend 2021-03-12T02:28:01.077 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-03-12T02:28:01.077 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-03-12T02:28:01.082 INFO:tasks.rgw.c2.client.1:Started 2021-03-12T02:28:01.082 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi077.front.sepia.ceph.com:8000/ 2021-03-12T02:28:01.083 DEBUG:teuthology.orchestra.run.smithi077:> curl http://smithi077.front.sepia.ceph.com:8000/ 2021-03-12T02:28:01.109 INFO:teuthology.orchestra.run.smithi077.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-12T02:28:01.110 INFO:teuthology.orchestra.run.smithi077.stderr: Dload Upload Total Spent Left Speed 2021-03-12T02:28:01.113 INFO:teuthology.orchestra.run.smithi077.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi077.front.sepia.ceph.com port 8000: Connection refused 2021-03-12T02:28:01.114 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-03-12T02:28:02.115 DEBUG:teuthology.orchestra.run.smithi077:> curl http://smithi077.front.sepia.ceph.com:8000/ 2021-03-12T02:28:02.131 INFO:teuthology.orchestra.run.smithi077.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-12T02:28:02.132 INFO:teuthology.orchestra.run.smithi077.stderr: Dload Upload Total Spent Left Speed 2021-03-12T02:28:02.139 INFO:teuthology.orchestra.run.smithi077.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2021-03-12T02:28:02.140 INFO:teuthology.orchestra.run.smithi077.stdout:anonymous 2021-03-12T02:28:02.140 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi077.front.sepia.ceph.com:8001/ 2021-03-12T02:28:02.141 DEBUG:teuthology.orchestra.run.smithi077:> curl http://smithi077.front.sepia.ceph.com:8001/ 2021-03-12T02:28:02.197 INFO:teuthology.orchestra.run.smithi077.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-12T02:28:02.197 INFO:teuthology.orchestra.run.smithi077.stderr: Dload Upload Total Spent Left Speed 2021-03-12T02:28:02.201 INFO:teuthology.orchestra.run.smithi077.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 35666 2021-03-12T02:28:02.203 INFO:teuthology.orchestra.run.smithi077.stdout:anonymous 2021-03-12T02:28:02.203 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi114.front.sepia.ceph.com:8000/ 2021-03-12T02:28:02.204 DEBUG:teuthology.orchestra.run.smithi114:> curl http://smithi114.front.sepia.ceph.com:8000/ 2021-03-12T02:28:02.238 INFO:teuthology.orchestra.run.smithi114.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-12T02:28:02.238 INFO:teuthology.orchestra.run.smithi114.stderr: Dload Upload Total Spent Left Speed 2021-03-12T02:28:02.242 INFO:teuthology.orchestra.run.smithi114.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 26750 0 --:--:-- --:--:-- --:--:-- 53500 2021-03-12T02:28:02.243 INFO:teuthology.orchestra.run.smithi114.stdout:anonymous 2021-03-12T02:28:02.243 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi114.front.sepia.ceph.com:8001/ 2021-03-12T02:28:02.244 DEBUG:teuthology.orchestra.run.smithi114:> curl http://smithi114.front.sepia.ceph.com:8001/ 2021-03-12T02:28:02.304 INFO:teuthology.orchestra.run.smithi114.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-12T02:28:02.304 INFO:teuthology.orchestra.run.smithi114.stderr: Dload Upload Total Spent Left Speed 2021-03-12T02:28:02.309 INFO:teuthology.orchestra.run.smithi114.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2021-03-12T02:28:02.310 INFO:teuthology.orchestra.run.smithi114.stdout:anonymous 2021-03-12T02:28:02.310 INFO:teuthology.run_tasks:Running task rgw-multisite... 2021-03-12T02:28:02.522 INFO:tasks.rgw_multisite:creating realm.. 2021-03-12T02:28:02.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:02.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:02.524 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:28:02.561 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:02.561 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:02.607 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"0428d020-efe8-4edc-a347-2def7c927539","name":"test-realm","current_period":"d9b128b7-cc24-4c7b-8b74-60a40f9eb5d0","epoch":1} 2021-03-12T02:28:02.607 INFO:tasks.rgw_multisite:creating master zonegroup.. 2021-03-12T02:28:02.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:02.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:02.609 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:28:02.686 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:02.686 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:02.710 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:28:02.708+0000 7f017f592080 -1 old format 2021-03-12T02:28:02.725 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}} 2021-03-12T02:28:02.726 INFO:tasks.rgw_multisite:creating master zone.. 2021-03-12T02:28:02.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--endpoints', 'http://smithi077.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:02.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--endpoints', 'http://smithi077.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:02.727 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key IMFN8A2KI1WM97OP --secret SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE --endpoints http://smithi077.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:28:02.815 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:02.815 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:02.877 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"IMFN8A2KI1WM97OP","secret_key":"SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"0428d020-efe8-4edc-a347-2def7c927539"} 2021-03-12T02:28:02.878 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2021-03-12T02:28:03.223 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'test-zone1.rgw.buckets.data' created 2021-03-12T02:28:03.234 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2021-03-12T02:28:04.244 INFO:teuthology.orchestra.run.smithi077.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2021-03-12T02:28:04.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:04.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:04.264 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:28:04.298 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:04.299 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:04.325 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"IMFN8A2KI1WM97OP","secret_key":"SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-12T02:28:04.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:04.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:04.326 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:28:04.407 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:04.407 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:12.467 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"0cf3ee11-29e6-4776-941b-50bda62adf82","epoch":1,"predecessor_uuid":"d9b128b7-cc24-4c7b-8b74-60a40f9eb5d0","sync_status":[],"period_map":{"id":"0cf3ee11-29e6-4776-941b-50bda62adf82","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","zones":[{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","val":2537029094}]},"master_zonegroup":"test-zonegroup","master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"0428d020-efe8-4edc-a347-2def7c927539","realm_name":"test-realm","realm_epoch":2} 2021-03-12T02:28:12.467 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-03-12T02:28:12.468 INFO:teuthology.orchestra.run:waiting for 300 2021-03-12T02:28:12.509 INFO:tasks.rgw.c1.client.0.smithi077.stdout:2021-03-12T02:28:12.508+0000 7fbca66f8700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 15065) UID: 0 2021-03-12T02:28:12.510 INFO:tasks.rgw.c1.client.0.smithi077.stdout:2021-03-12T02:28:12.508+0000 7fbcbd5bb0c0 -1 shutting down 2021-03-12T02:28:15.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-03-12T02:28:18.572 INFO:tasks.rgw.c1.client.0:Stopped 2021-03-12T02:28:18.573 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-03-12T02:28:18.574 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-03-12T02:28:18.577 INFO:tasks.rgw.c1.client.0:Started 2021-03-12T02:28:18.578 DEBUG:teuthology.orchestra.run.smithi077:> curl http://smithi077.front.sepia.ceph.com:8000 2021-03-12T02:28:18.595 INFO:teuthology.orchestra.run.smithi077.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-12T02:28:18.596 INFO:teuthology.orchestra.run.smithi077.stderr: Dload Upload Total Spent Left Speed 2021-03-12T02:28:18.599 INFO:teuthology.orchestra.run.smithi077.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi077.front.sepia.ceph.com port 8000: Connection refused 2021-03-12T02:28:18.600 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-03-12T02:28:19.601 DEBUG:teuthology.orchestra.run.smithi077:> curl http://smithi077.front.sepia.ceph.com:8000 2021-03-12T02:28:19.618 INFO:teuthology.orchestra.run.smithi077.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-12T02:28:19.618 INFO:teuthology.orchestra.run.smithi077.stderr: Dload Upload Total Spent Left Speed 2021-03-12T02:28:19.622 INFO:teuthology.orchestra.run.smithi077.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2021-03-12T02:28:19.623 INFO:teuthology.orchestra.run.smithi077.stdout:anonymous 2021-03-12T02:28:19.624 INFO:tasks.rgw_multisite:creating admin user.. 2021-03-12T02:28:19.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:19.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:19.625 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key IMFN8A2KI1WM97OP --secret SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:28:19.710 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:19.710 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:19.865 INFO:teuthology.orchestra.run.smithi077.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"IMFN8A2KI1WM97OP","secret_key":"SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":"true","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2021-03-12T02:28:19.891 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-03-12T02:28:19.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi077.front.sepia.ceph.com:8000', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:19.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi077.front.sepia.ceph.com:8000', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:19.892 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi077.front.sepia.ceph.com:8000 --access-key IMFN8A2KI1WM97OP --secret SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:28:19.936 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:19.936 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:20.042 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"0428d020-efe8-4edc-a347-2def7c927539","name":"test-realm","current_period":"0cf3ee11-29e6-4776-941b-50bda62adf82","epoch":2} 2021-03-12T02:28:20.043 INFO:tasks.rgw_multisite:creating zone.. 2021-03-12T02:28:20.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--endpoints', 'http://smithi114.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:20.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--endpoints', 'http://smithi114.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:20.044 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key IMFN8A2KI1WM97OP --secret SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE --endpoints http://smithi114.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:28:20.089 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:20.089 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:20.144 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:20.136+0000 7f4aa600e080 0 failed reading obj info from .rgw.root:zone_info.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e: (2) No such file or directory 2021-03-12T02:28:20.144 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:20.136+0000 7f4aa600e080 0 WARNING: could not read zone params for zone id=6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e name=test-zone1 2021-03-12T02:28:20.156 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"IMFN8A2KI1WM97OP","secret_key":"SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"0428d020-efe8-4edc-a347-2def7c927539"} 2021-03-12T02:28:20.157 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2021-03-12T02:28:21.460 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'test-zone2.rgw.buckets.data' created 2021-03-12T02:28:21.480 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2021-03-12T02:28:21.579 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-03-12T02:28:22.474 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2021-03-12T02:28:22.500 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:22.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:22.501 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:28:22.535 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:22.535 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:22.557 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"IMFN8A2KI1WM97OP","secret_key":"SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-12T02:28:22.558 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:22.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:22.559 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:28:22.638 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:22.639 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:30.642 INFO:teuthology.orchestra.run.smithi114.stderr:Sending period to new master zone 6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e 2021-03-12T02:28:31.720 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"0cf3ee11-29e6-4776-941b-50bda62adf82","epoch":2,"predecessor_uuid":"d9b128b7-cc24-4c7b-8b74-60a40f9eb5d0","sync_status":[],"period_map":{"id":"0cf3ee11-29e6-4776-941b-50bda62adf82","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","zones":[{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","endpoints":["http://smithi114.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"374a14e6-b245-4495-acdf-8a1b3996ba1d","val":3185083755},{"key":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","val":2537029094}]},"master_zonegroup":"test-zonegroup","master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"0428d020-efe8-4edc-a347-2def7c927539","realm_name":"test-realm","realm_epoch":2} 2021-03-12T02:28:31.721 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-03-12T02:28:31.721 INFO:teuthology.orchestra.run:waiting for 300 2021-03-12T02:28:31.764 INFO:tasks.rgw.c2.client.0.smithi114.stdout:2021-03-12T02:28:31.760+0000 7fad31ba6700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 15114) UID: 0 2021-03-12T02:28:31.765 INFO:tasks.rgw.c2.client.0.smithi114.stdout:2021-03-12T02:28:31.760+0000 7fad4926a0c0 -1 shutting down 2021-03-12T02:28:35.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-03-12T02:28:37.823 INFO:tasks.rgw.c2.client.0:Stopped 2021-03-12T02:28:37.823 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-03-12T02:28:37.824 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-03-12T02:28:37.826 INFO:tasks.rgw.c2.client.0:Started 2021-03-12T02:28:37.827 DEBUG:teuthology.orchestra.run.smithi114:> curl http://smithi114.front.sepia.ceph.com:8000 2021-03-12T02:28:37.856 INFO:teuthology.orchestra.run.smithi114.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-12T02:28:37.856 INFO:teuthology.orchestra.run.smithi114.stderr: Dload Upload Total Spent Left Speed 2021-03-12T02:28:37.859 INFO:teuthology.orchestra.run.smithi114.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi114.front.sepia.ceph.com port 8000: Connection refused 2021-03-12T02:28:37.860 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-03-12T02:28:38.861 DEBUG:teuthology.orchestra.run.smithi114:> curl http://smithi114.front.sepia.ceph.com:8000 2021-03-12T02:28:38.875 INFO:teuthology.orchestra.run.smithi114.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-12T02:28:38.875 INFO:teuthology.orchestra.run.smithi114.stderr: Dload Upload Total Spent Left Speed 2021-03-12T02:28:38.880 INFO:teuthology.orchestra.run.smithi114.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2021-03-12T02:28:38.881 INFO:teuthology.orchestra.run.smithi114.stdout:anonymous 2021-03-12T02:28:38.881 INFO:tasks.rgw_multisite:creating zone.. 2021-03-12T02:28:38.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--endpoints', 'http://smithi077.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:38.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--endpoints', 'http://smithi077.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:38.882 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key IMFN8A2KI1WM97OP --secret SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE --endpoints http://smithi077.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:28:38.927 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:38.927 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:38.977 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:28:38.976+0000 7f724a2e5080 0 failed reading obj info from .rgw.root:zone_info.374a14e6-b245-4495-acdf-8a1b3996ba1d: (2) No such file or directory 2021-03-12T02:28:38.978 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:28:38.976+0000 7f724a2e5080 0 WARNING: could not read zone params for zone id=374a14e6-b245-4495-acdf-8a1b3996ba1d name=test-zone2 2021-03-12T02:28:38.989 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"IMFN8A2KI1WM97OP","secret_key":"SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":"0428d020-efe8-4edc-a347-2def7c927539"} 2021-03-12T02:28:38.990 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2021-03-12T02:28:40.224 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'test-zone3.rgw.buckets.data' created 2021-03-12T02:28:40.236 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2021-03-12T02:28:41.228 INFO:teuthology.orchestra.run.smithi077.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2021-03-12T02:28:41.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:41.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:41.244 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:28:41.284 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:41.284 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:41.308 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"IMFN8A2KI1WM97OP","secret_key":"SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-12T02:28:41.309 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:41.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:41.309 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:28:41.390 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:41.391 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:41.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2021-03-12T02:28:49.415 INFO:teuthology.orchestra.run.smithi077.stderr:Sending period to new master zone 6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e 2021-03-12T02:28:50.493 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"0cf3ee11-29e6-4776-941b-50bda62adf82","epoch":3,"predecessor_uuid":"d9b128b7-cc24-4c7b-8b74-60a40f9eb5d0","sync_status":[],"period_map":{"id":"0cf3ee11-29e6-4776-941b-50bda62adf82","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","zones":[{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","endpoints":["http://smithi114.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","name":"test-zone3","endpoints":["http://smithi077.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"374a14e6-b245-4495-acdf-8a1b3996ba1d","val":3185083755},{"key":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","val":2537029094},{"key":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","val":3019679815}]},"master_zonegroup":"test-zonegroup","master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"0428d020-efe8-4edc-a347-2def7c927539","realm_name":"test-realm","realm_epoch":2} 2021-03-12T02:28:50.494 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-03-12T02:28:50.494 INFO:teuthology.orchestra.run:waiting for 300 2021-03-12T02:28:50.537 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:28:50.536+0000 7f378cdcb700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 15068) UID: 0 2021-03-12T02:28:50.538 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:28:50.536+0000 7f37a448f0c0 -1 shutting down 2021-03-12T02:28:55.717 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-03-12T02:28:56.597 INFO:tasks.rgw.c1.client.1:Stopped 2021-03-12T02:28:56.597 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-03-12T02:28:56.598 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-03-12T02:28:56.601 INFO:tasks.rgw.c1.client.1:Started 2021-03-12T02:28:56.601 DEBUG:teuthology.orchestra.run.smithi077:> curl http://smithi077.front.sepia.ceph.com:8001 2021-03-12T02:28:56.618 INFO:teuthology.orchestra.run.smithi077.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-12T02:28:56.618 INFO:teuthology.orchestra.run.smithi077.stderr: Dload Upload Total Spent Left Speed 2021-03-12T02:28:56.622 INFO:teuthology.orchestra.run.smithi077.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi077.front.sepia.ceph.com port 8001: Connection refused 2021-03-12T02:28:56.622 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-03-12T02:28:57.624 DEBUG:teuthology.orchestra.run.smithi077:> curl http://smithi077.front.sepia.ceph.com:8001 2021-03-12T02:28:57.641 INFO:teuthology.orchestra.run.smithi077.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-12T02:28:57.641 INFO:teuthology.orchestra.run.smithi077.stderr: Dload Upload Total Spent Left Speed 2021-03-12T02:28:57.645 INFO:teuthology.orchestra.run.smithi077.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2021-03-12T02:28:57.648 INFO:teuthology.orchestra.run.smithi077.stdout:anonymous 2021-03-12T02:28:57.648 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-03-12T02:28:57.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi077.front.sepia.ceph.com:8000', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--url', 'http://smithi077.front.sepia.ceph.com:8000', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:57.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi077.front.sepia.ceph.com:8000', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--url', 'http://smithi077.front.sepia.ceph.com:8000', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:57.649 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi077.front.sepia.ceph.com:8000 --access-key IMFN8A2KI1WM97OP --secret SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE --url http://smithi077.front.sepia.ceph.com:8000 --access-key IMFN8A2KI1WM97OP --secret SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:28:57.697 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:57.697 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:57.736 INFO:tasks.rgw.c2.client.0.smithi114.stdout:2021-03-12T02:28:57.732+0000 7f3479602700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.0428d020-efe8-4edc-a347-2def7c927539.control err=-107 2021-03-12T02:28:57.765 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"0428d020-efe8-4edc-a347-2def7c927539","name":"test-realm","current_period":"0cf3ee11-29e6-4776-941b-50bda62adf82","epoch":2} 2021-03-12T02:28:57.766 INFO:tasks.rgw_multisite:creating zone.. 2021-03-12T02:28:57.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--endpoints', 'http://smithi114.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:57.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--endpoints', 'http://smithi114.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:57.767 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key IMFN8A2KI1WM97OP --secret SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE --endpoints http://smithi114.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:28:57.820 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:57.820 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:57.874 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:57.868+0000 7f6b0bb0c080 0 failed reading obj info from .rgw.root:zone_info.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e: (2) No such file or directory 2021-03-12T02:28:57.874 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:57.868+0000 7f6b0bb0c080 0 WARNING: could not read zone params for zone id=6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e name=test-zone1 2021-03-12T02:28:57.875 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:57.872+0000 7f6b0bb0c080 0 failed reading obj info from .rgw.root:zone_info.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107: (2) No such file or directory 2021-03-12T02:28:57.875 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:28:57.872+0000 7f6b0bb0c080 0 WARNING: could not read zone params for zone id=e98c2a8b-a28f-47cd-a0ca-2777b6b4f107 name=test-zone3 2021-03-12T02:28:57.886 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"IMFN8A2KI1WM97OP","secret_key":"SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"0428d020-efe8-4edc-a347-2def7c927539"} 2021-03-12T02:28:57.886 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2021-03-12T02:28:58.467 INFO:teuthology.orchestra.run.smithi114.stderr:pool 'test-zone4.rgw.buckets.data' created 2021-03-12T02:28:58.490 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2021-03-12T02:28:59.488 INFO:teuthology.orchestra.run.smithi114.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2021-03-12T02:28:59.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:59.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:59.517 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:28:59.558 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:59.558 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:28:59.583 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"IMFN8A2KI1WM97OP","secret_key":"SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":""} 2021-03-12T02:28:59.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:59.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:28:59.585 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:28:59.665 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:28:59.665 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:29:01.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-03-12T02:29:07.666 INFO:teuthology.orchestra.run.smithi114.stderr:Sending period to new master zone 6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e 2021-03-12T02:29:08.936 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"0cf3ee11-29e6-4776-941b-50bda62adf82","epoch":4,"predecessor_uuid":"d9b128b7-cc24-4c7b-8b74-60a40f9eb5d0","sync_status":[],"period_map":{"id":"0cf3ee11-29e6-4776-941b-50bda62adf82","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","zones":[{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","endpoints":["http://smithi114.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","name":"test-zone4","endpoints":["http://smithi114.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","name":"test-zone3","endpoints":["http://smithi077.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"374a14e6-b245-4495-acdf-8a1b3996ba1d","val":3185083755},{"key":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","val":2537029094},{"key":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","val":3794530578},{"key":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","val":3019679815}]},"master_zonegroup":"test-zonegroup","master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"0428d020-efe8-4edc-a347-2def7c927539","realm_name":"test-realm","realm_epoch":2} 2021-03-12T02:29:08.937 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-03-12T02:29:08.937 INFO:teuthology.orchestra.run:waiting for 300 2021-03-12T02:29:08.979 INFO:tasks.rgw.c2.client.1.smithi114.stdout:2021-03-12T02:29:08.976+0000 7fcfe37fe700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 15116) UID: 0 2021-03-12T02:29:08.980 INFO:tasks.rgw.c2.client.1.smithi114.stdout:2021-03-12T02:29:08.976+0000 7fd0075640c0 -1 shutting down 2021-03-12T02:29:10.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-03-12T02:29:15.039 INFO:tasks.rgw.c2.client.1:Stopped 2021-03-12T02:29:15.040 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-03-12T02:29:15.041 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-03-12T02:29:15.048 INFO:tasks.rgw.c2.client.1:Started 2021-03-12T02:29:15.049 DEBUG:teuthology.orchestra.run.smithi114:> curl http://smithi114.front.sepia.ceph.com:8001 2021-03-12T02:29:15.076 INFO:teuthology.orchestra.run.smithi114.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-12T02:29:15.077 INFO:teuthology.orchestra.run.smithi114.stderr: Dload Upload Total Spent Left Speed 2021-03-12T02:29:15.079 INFO:teuthology.orchestra.run.smithi114.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi114.front.sepia.ceph.com port 8001: Connection refused 2021-03-12T02:29:15.080 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-03-12T02:29:15.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2021-03-12T02:29:16.082 DEBUG:teuthology.orchestra.run.smithi114:> curl http://smithi114.front.sepia.ceph.com:8001 2021-03-12T02:29:16.100 INFO:teuthology.orchestra.run.smithi114.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-12T02:29:16.100 INFO:teuthology.orchestra.run.smithi114.stderr: Dload Upload Total Spent Left Speed 2021-03-12T02:29:16.104 INFO:teuthology.orchestra.run.smithi114.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 26800 0 --:--:-- --:--:-- --:--:-- 26800 2021-03-12T02:29:16.105 INFO:teuthology.orchestra.run.smithi114.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-00604ad1fc-10e6-test-zone4","HostId":"10e6-test-zone4-test-zonegroup"} 2021-03-12T02:29:16.105 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2021-03-12T02:29:16.106 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2021-03-12T02:29:16.169 INFO:tasks.rgw_multisite_tests:creating test user.. 2021-03-12T02:29:16.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:29:16.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:29:16.171 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:29:16.215 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:29:16.215 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:29:16.399 INFO:teuthology.orchestra.run.smithi077.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"2705WDYYP7W490AQXJ3H","secret_key":"uKG3SrE6DNCmaGHq4713hmZSyfqAC5lZVHWQiitO"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2021-03-12T02:29:16.425 INFO:rgw_multi.tests:meta checkpoint 2021-03-12T02:29:16.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:16.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:16.428 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:29:16.504 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:29:16.504 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:29:16.668 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:29:16.703 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:29:16.703 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:29:16.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:16.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:16.705 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:29:16.745 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:29:16.746 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:29:16.941 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:29:16.942 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:29:21.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:21.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:21.944 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:29:21.986 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:29:21.987 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:29:22.176 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:29:22.177 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:29:27.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:27.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:27.179 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:29:27.222 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:29:27.223 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:29:27.424 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:29:27.424 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:29:32.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:32.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:32.427 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:29:32.467 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:29:32.468 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:29:32.654 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:29:32.655 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:29:37.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:37.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:37.657 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:29:37.700 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:29:37.701 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:29:37.903 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:29:37.904 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:29:42.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:42.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:42.906 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:29:42.949 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:29:42.949 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:29:43.155 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:29:43.156 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:29:48.157 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:48.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:48.158 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:29:48.199 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:29:48.200 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:29:48.390 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:29:48.390 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:29:53.392 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:53.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:53.393 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:29:53.434 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:29:53.435 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:29:53.614 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:29:53.614 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:29:58.615 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:58.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:29:58.616 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:29:58.656 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:29:58.657 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:29:58.857 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:29:58.857 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:30:03.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:03.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:03.860 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:30:03.900 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:30:03.901 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:30:04.083 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:30:04.084 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:30:09.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:09.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:09.089 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:30:09.131 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:30:09.131 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:30:09.320 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:30:09.320 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:30:14.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:14.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:14.323 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:30:14.366 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:30:14.367 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:30:14.560 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:30:14.561 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:30:19.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:19.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:19.563 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:30:19.602 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:30:19.603 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:30:19.794 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:30:19.795 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:30:24.797 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:24.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:24.798 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:30:24.838 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:30:24.839 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:30:25.024 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:30:25.025 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:30:30.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:30.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:30.028 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:30:30.068 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:30:30.068 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:30:30.248 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:30:30.249 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:30:35.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:35.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:35.251 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:30:35.292 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:30:35.293 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:30:35.497 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:30:35.498 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:30:40.499 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:40.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:40.500 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:30:40.541 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:30:40.542 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:30:40.725 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:30:40.726 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:30:45.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:45.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:45.728 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:30:45.769 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:30:45.770 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:30:45.952 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:30:45.953 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:30:45.954 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:30:45.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:45.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:45.956 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:30:45.998 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:30:45.998 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:30:46.173 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:30:46.173 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:30:51.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:51.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:51.176 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:30:51.214 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:30:51.215 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:30:51.420 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:30:51.421 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:30:56.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:56.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:30:56.423 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:30:56.464 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:30:56.464 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:30:56.659 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:30:56.660 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:31:01.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:01.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:01.662 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:01.703 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:01.703 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:01.898 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:01.899 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:31:06.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:06.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:06.901 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:06.941 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:06.941 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:07.134 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:07.135 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1615516156.395861_98.1 target= 2021-03-12T02:31:12.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:12.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:12.137 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:12.178 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:12.179 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:12.381 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:12.382 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:31:12.383 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:31:12.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:12.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:12.384 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:12.425 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:12.425 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:12.682 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:12.683 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:31:12.684 INFO:rgw_multi.tests:meta checkpoint 2021-03-12T02:31:12.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:12.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:12.686 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:12.726 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:12.727 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:12.905 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:31:12.941 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:31:12.942 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:31:12.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:12.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:12.943 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:12.985 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:12.985 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:13.172 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:13.173 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:31:13.173 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:31:13.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:13.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:13.174 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:13.218 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:13.219 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:13.414 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:13.415 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:31:13.416 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:31:13.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:13.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:13.417 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:13.457 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:13.458 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:13.716 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:13.717 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:31:13.717 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2021-03-12T02:31:13.725 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-1 2021-03-12T02:31:17.258 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-2 2021-03-12T02:31:20.556 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-3 2021-03-12T02:31:20.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:20.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:20.607 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:20.642 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:20.643 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:20.797 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:31:20.829 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516280.593132_95.1","last_update":"2021-03-12T02:31:20.593132Z"}] 2021-03-12T02:31:20.829 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:31:20.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:20.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:20.831 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:20.874 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:20.874 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:21.057 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516280.582981_92.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:20.582981Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:21.058 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1615516280.593132_95.1 target=1_1615516280.582981_92.1 2021-03-12T02:31:26.059 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:26.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:26.061 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:26.103 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:26.103 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:26.301 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516280.593132_95.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:20.593132Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:26.302 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:31:26.302 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:31:26.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:26.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:26.305 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:26.345 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:26.345 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:26.559 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516280.593132_95.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:20.593132Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:26.560 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:31:26.561 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:31:26.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:26.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:26.565 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:26.607 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:26.607 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:26.859 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516280.593132_95.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:20.593132Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:26.860 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:31:28.226 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2021-03-12T02:31:28.848 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-4 2021-03-12T02:31:29.401 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-5 2021-03-12T02:31:29.936 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-6 2021-03-12T02:31:30.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:30.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:30.473 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:30.512 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:30.512 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:30.672 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:31:30.699 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516290.459112_108.1","last_update":"2021-03-12T02:31:30.459112Z"}] 2021-03-12T02:31:30.699 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:31:30.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:30.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:30.701 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:30.739 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:30.740 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:30.920 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516290.459112_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:30.459112Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:30.921 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:31:30.921 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:31:30.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:30.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:30.923 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:30.965 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:30.966 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:31.150 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516290.459112_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:30.459112Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:31.150 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:31:31.151 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:31:31.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:31.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:31.152 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:31.192 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:31.193 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:31.450 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516290.459112_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:30.459112Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:31.450 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:31:33.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:33.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:33.327 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:33.364 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:33.365 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:33.523 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:31:33.554 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516293.216631_145.1","last_update":"2021-03-12T02:31:33.216631Z"}] 2021-03-12T02:31:33.555 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:31:33.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:33.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:33.556 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:33.595 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:33.597 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:33.784 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516292.676123_109.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:32.676123Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:33.785 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1615516293.216631_145.1 target=1_1615516292.676123_109.1 2021-03-12T02:31:38.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:38.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:38.788 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:38.828 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:38.828 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:39.028 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516293.216631_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:33.216631Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:39.029 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:31:39.029 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:31:39.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:39.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:39.031 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:39.077 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:39.077 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:39.269 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516293.216631_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:33.216631Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:39.270 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:31:39.270 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:31:39.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:39.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:39.272 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:39.314 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:39.315 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:39.568 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516293.216631_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:33.216631Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:39.569 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:31:40.474 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2021-03-12T02:31:40.475 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-7 2021-03-12T02:31:40.505 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-8 2021-03-12T02:31:40.559 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-9 2021-03-12T02:31:40.610 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:40.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:40.611 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:40.644 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:40.645 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:40.809 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:31:40.842 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516300.595034_159.1","last_update":"2021-03-12T02:31:40.595034Z"}] 2021-03-12T02:31:40.842 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:31:40.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:40.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:40.844 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:40.886 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:40.887 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:41.107 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516300.595034_159.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:40.595034Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:41.108 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:31:41.109 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:31:41.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:41.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:41.110 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:41.152 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:41.153 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:41.352 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516293.216631_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:33.216631Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:41.354 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1615516300.595034_159.1 target=1_1615516293.216631_145.1 2021-03-12T02:31:46.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:46.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:46.356 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:46.397 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:46.398 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:46.596 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516300.595034_159.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:40.595034Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:46.596 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:31:46.597 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:31:46.597 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:46.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:46.598 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:46.638 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:46.638 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:46.896 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516300.595034_159.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:40.595034Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:46.896 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:31:49.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:49.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:49.619 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:49.655 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:49.656 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:49.831 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:31:49.860 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516309.597935_168.1","last_update":"2021-03-12T02:31:49.597935Z"}] 2021-03-12T02:31:49.860 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:31:49.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:49.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:49.862 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:49.904 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:49.905 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:50.099 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516300.595034_159.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:40.595034Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:50.099 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1615516309.597935_168.1 target=1_1615516300.595034_159.1 2021-03-12T02:31:55.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:55.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:55.102 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:55.143 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:55.144 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:55.344 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:55.344 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:31:55.345 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:31:55.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:55.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:55.346 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:55.387 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:55.387 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:55.601 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:55.601 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:31:55.602 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:31:55.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:55.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:55.603 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:55.644 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:55.644 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:55.902 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:55.902 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:31:57.301 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2021-03-12T02:31:57.302 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-10 2021-03-12T02:31:57.526 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-11 2021-03-12T02:31:57.951 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-12 2021-03-12T02:31:58.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:58.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:58.584 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:58.623 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:58.623 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:58.788 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:31:58.817 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516317.983732_69.1","last_update":"2021-03-12T02:31:57.983732Z"},{"marker":"1_1615516309.597935_168.1","last_update":"2021-03-12T02:31:49.597935Z"}] 2021-03-12T02:31:58.818 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:31:58.818 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:58.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:58.819 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:58.860 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:58.861 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:31:59.063 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516317.983732_69.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:57.983732Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:31:59.950 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:31:59.950 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:31:59.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:59.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:31:59.952 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:31:59.993 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:31:59.994 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:00.209 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516317.983732_69.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:57.983732Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:32:00.210 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:32:00.210 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:32:00.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:00.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:00.212 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:00.256 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:00.256 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:00.507 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516317.983732_69.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:57.983732Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:32:00.508 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:32:00.508 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:00.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:00.509 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:00.549 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:00.744 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.13.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.7.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.24.5"},{"key":8,"val":"00000000001.13.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:00.744 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-10 2021-03-12T02:32:00.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:00.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:00.746 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:00.789 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:00.790 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:00.991 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.24.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:00.991 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-10 2021-03-12T02:32:00.992 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-10 zones={test-zone2, test-zone1} 2021-03-12T02:32:01.484 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:32:01.491 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-12T02:32:01.492 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-12T02:32:01.492 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-12T02:32:01.493 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:01.493 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:32:01.500 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-12T02:32:01.500 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-12T02:32:01.500 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-12T02:32:01.501 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:01.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:01.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:01.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:01.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:01.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:01.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:02.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:02.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:02.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:02.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:02.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:02.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:02.627 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-10 zones={test-zone2, test-zone1} 2021-03-12T02:32:02.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:02.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:02.629 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:02.668 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:02.669 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:02.870 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.13.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.7.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.24.5"},{"key":8,"val":"00000000001.13.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:02.871 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-10 2021-03-12T02:32:02.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:02.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:02.873 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:02.955 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:02.956 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:03.155 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.24.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:03.156 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-10 2021-03-12T02:32:03.157 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-10 zones={test-zone3, test-zone1} 2021-03-12T02:32:03.333 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:32:03.340 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-12T02:32:03.340 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-12T02:32:03.341 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-12T02:32:03.341 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:03.341 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:32:03.346 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-12T02:32:03.346 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-12T02:32:03.347 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-12T02:32:03.347 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:03.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:03.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:03.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:03.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:03.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:03.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:03.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:03.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:04.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:04.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:04.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:04.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:04.462 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-10 zones={test-zone3, test-zone1} 2021-03-12T02:32:04.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:04.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:04.464 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:04.504 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:04.504 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:04.706 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.13.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.7.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.24.5"},{"key":8,"val":"00000000001.13.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:04.707 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-10 2021-03-12T02:32:04.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:04.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:04.708 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:04.753 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:04.754 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:05.016 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.13.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.7.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.24.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.13.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:05.017 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-10 2021-03-12T02:32:05.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:05.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:05.018 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:05.099 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:05.100 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:05.286 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.4.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.6.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.10.5"},{"key":8,"val":"00000000001.12.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:05.286 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-11 2021-03-12T02:32:05.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:05.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:05.288 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:05.331 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:05.332 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:05.537 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.4.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.10.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:05.538 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-11 2021-03-12T02:32:05.539 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-11 zones={test-zone1, test-zone2} 2021-03-12T02:32:07.104 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:32:07.113 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-12T02:32:07.113 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-12T02:32:07.113 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-12T02:32:07.114 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:07.114 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:32:07.120 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-12T02:32:07.121 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-12T02:32:07.121 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-12T02:32:07.122 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:07.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:07.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:07.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:07.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:07.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:07.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:07.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:07.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:07.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:07.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:08.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:08.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:08.217 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-11 zones={test-zone1, test-zone2} 2021-03-12T02:32:08.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:08.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:08.218 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:08.256 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:08.257 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:08.448 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.4.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.6.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.10.5"},{"key":8,"val":"00000000001.12.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:08.448 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-11 2021-03-12T02:32:08.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:08.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:08.450 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:08.494 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:08.495 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:08.708 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.4.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.10.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:08.708 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-11 2021-03-12T02:32:08.709 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-11 zones={test-zone3, test-zone2} 2021-03-12T02:32:09.630 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:32:09.638 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-12T02:32:09.638 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-12T02:32:09.639 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-12T02:32:09.640 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:09.641 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:32:09.647 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-12T02:32:09.648 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-12T02:32:09.648 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-12T02:32:09.649 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:09.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:09.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:09.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:09.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:09.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:10.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:10.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:10.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:10.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:10.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:10.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:10.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:10.772 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-11 zones={test-zone3, test-zone2} 2021-03-12T02:32:10.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:10.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:10.774 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:10.816 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:10.816 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:11.017 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.5.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.11.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.7.5"},{"key":8,"val":"00000000001.5.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:11.017 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-12 2021-03-12T02:32:11.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:11.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:11.019 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:11.101 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:11.101 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:11.304 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.5.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.11.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.5.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:11.305 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-12 2021-03-12T02:32:11.305 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-12 zones={test-zone1, test-zone3} 2021-03-12T02:32:11.581 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:32:11.587 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-12T02:32:11.587 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-12T02:32:11.588 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-12T02:32:11.589 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:11.589 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:32:11.593 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-12T02:32:11.594 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-12T02:32:11.594 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-12T02:32:11.595 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:11.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:11.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:11.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:11.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:11.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:12.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:12.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:12.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:12.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:12.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:12.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:12.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:12.737 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-12 zones={test-zone1, test-zone3} 2021-03-12T02:32:12.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:12.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:12.739 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:12.778 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:12.779 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:12.981 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.5.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.11.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.7.5"},{"key":8,"val":"00000000001.5.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:12.981 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-12 2021-03-12T02:32:12.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:12.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:12.983 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-12 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:13.027 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:13.027 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:13.216 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.5.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.11.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.5.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:13.216 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-12 2021-03-12T02:32:13.217 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-12 zones={test-zone2, test-zone3} 2021-03-12T02:32:14.647 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:32:14.654 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-12T02:32:14.655 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-12T02:32:14.655 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-12T02:32:14.655 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:14.656 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:32:14.664 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-12T02:32:14.664 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-12T02:32:14.665 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-12T02:32:14.665 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:14.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:14.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:14.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-12T02:32:14.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:15.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:15.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-12T02:32:15.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:15.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:15.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-12T02:32:15.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:15.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:15.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:15.798 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-12 zones={test-zone2, test-zone3} 2021-03-12T02:32:15.799 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2021-03-12T02:32:15.800 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-13 2021-03-12T02:32:16.002 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-14 2021-03-12T02:32:16.049 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-15 2021-03-12T02:32:16.247 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:16.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:16.248 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:16.287 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:16.287 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:16.439 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:32:16.467 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516336.084220_97.1","last_update":"2021-03-12T02:32:16.084220Z"},{"marker":"1_1615516309.597935_168.1","last_update":"2021-03-12T02:31:49.597935Z"}] 2021-03-12T02:32:16.467 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:32:16.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:16.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:16.469 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:16.508 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:16.508 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:16.701 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516317.983732_69.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:57.983732Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:32:16.701 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615516336.084220_97.1 target=1_1615516317.983732_69.1 2021-03-12T02:32:17.136 INFO:tasks.rgw.c2.client.0.smithi114.stdout:2021-03-12T02:32:17.136+0000 7f367fff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:32:17.137 INFO:tasks.rgw.c2.client.1.smithi114.stdout:2021-03-12T02:32:17.136+0000 7fe2557f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:32:17.138 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:32:17.140+0000 7f1e657fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:32:17.138 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:32:17.140+0000 7f1e5dfeb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:32:17.140 INFO:tasks.rgw.c2.client.1.smithi114.stdout:2021-03-12T02:32:17.140+0000 7fe257ff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:32:17.141 INFO:tasks.rgw.c2.client.0.smithi114.stdout:2021-03-12T02:32:17.140+0000 7f367dff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:32:21.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:21.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:21.704 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:21.744 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:21.744 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:21.922 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516336.084220_97.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:32:16.084220Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:32:21.923 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:32:21.923 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:32:21.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:21.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:21.925 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:21.967 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:21.967 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:22.152 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516336.084220_97.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:32:16.084220Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:32:22.152 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:32:22.153 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:32:22.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:22.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:22.154 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:22.195 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:22.195 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:22.453 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516336.084220_97.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:32:16.084220Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:32:22.453 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:32:22.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:22.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:22.455 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:22.498 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:22.498 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:22.683 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.15.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:22.684 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-13 2021-03-12T02:32:22.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:22.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:22.686 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:22.729 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:22.729 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:22.914 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:22.915 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-13 2021-03-12T02:32:22.915 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-13 zones={test-zone2, test-zone1} 2021-03-12T02:32:23.398 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:32:23.404 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:23.404 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:32:23.455 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:23.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:23.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:23.671 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:23.778 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-13 zones={test-zone2, test-zone1} 2021-03-12T02:32:23.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:23.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:23.780 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:23.819 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:23.819 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:24.005 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.15.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:24.006 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-13 2021-03-12T02:32:24.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:24.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:24.007 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:24.093 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:24.094 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:24.286 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:24.286 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-13 2021-03-12T02:32:24.287 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-13 zones={test-zone3, test-zone1} 2021-03-12T02:32:25.778 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:32:25.786 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:25.786 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:32:25.835 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:25.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:25.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:26.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:26.164 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-13 zones={test-zone3, test-zone1} 2021-03-12T02:32:26.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:26.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:26.165 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:26.205 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:26.206 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:26.399 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.15.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:26.400 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-13 2021-03-12T02:32:26.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:26.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:26.401 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:26.446 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:26.447 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:26.703 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.15.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:26.703 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-13 2021-03-12T02:32:26.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:26.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:26.705 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:26.786 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:26.787 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:26.959 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.9.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:26.960 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-14 2021-03-12T02:32:26.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:26.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:26.963 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:27.009 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:27.009 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:27.212 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:27.212 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-14 2021-03-12T02:32:27.213 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-14 zones={test-zone1, test-zone2} 2021-03-12T02:32:28.608 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:32:28.616 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:28.616 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:32:28.663 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:28.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:28.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:28.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:29.000 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-14 zones={test-zone1, test-zone2} 2021-03-12T02:32:29.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:29.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:29.002 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:29.041 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:29.041 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:29.217 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.9.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:29.218 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-14 2021-03-12T02:32:29.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:29.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:29.219 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:29.265 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:29.265 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:29.459 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:29.459 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-14 2021-03-12T02:32:29.460 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-14 zones={test-zone3, test-zone2} 2021-03-12T02:32:31.407 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:32:31.417 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:31.418 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:32:31.471 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:31.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:31.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:31.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:31.805 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-14 zones={test-zone3, test-zone2} 2021-03-12T02:32:31.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:31.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:31.806 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:31.848 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:31.849 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:32.035 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.8.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:32.036 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-15 2021-03-12T02:32:32.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:32.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:32.037 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:32.122 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:32.123 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:32.325 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:32.325 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-15 2021-03-12T02:32:32.326 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-15 zones={test-zone1, test-zone3} 2021-03-12T02:32:32.471 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:32:32.523 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:32.523 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:32:32.530 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:32.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:32.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:32.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:32.868 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-15 zones={test-zone1, test-zone3} 2021-03-12T02:32:32.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:32.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:32.870 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:32.909 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:32.910 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:33.097 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.8.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:33.097 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-15 2021-03-12T02:32:33.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:33.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:33.099 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:33.145 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:33.145 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:33.341 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:33.342 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-15 2021-03-12T02:32:33.342 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-15 zones={test-zone2, test-zone3} 2021-03-12T02:32:34.989 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:32:35.866 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:35.866 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:32:35.875 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-12T02:32:35.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:35.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:36.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-12T02:32:36.202 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-15 zones={test-zone2, test-zone3} 2021-03-12T02:32:36.434 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:36.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:36.435 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:36.473 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:36.658 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.17.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:36.659 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-13 2021-03-12T02:32:36.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:36.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:36.661 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:36.704 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:36.704 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:36.906 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:36.907 WARNING:rgw_multi.tests:bucket bbucnh-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.17.5 target=00000000001.15.5 2021-03-12T02:32:41.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:41.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:41.909 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:41.956 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:41.956 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:42.143 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.17.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:42.143 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-13 2021-03-12T02:32:42.144 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-13 zones={test-zone2, test-zone1} 2021-03-12T02:32:43.548 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:32:43.553 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:32:43.658 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-13 zones={test-zone2, test-zone1} 2021-03-12T02:32:43.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:43.660 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:43.660 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:43.703 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:43.703 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:43.896 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.17.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:43.896 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-13 2021-03-12T02:32:43.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:43.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:43.898 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:43.983 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:43.983 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:44.177 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.17.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:44.178 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-13 2021-03-12T02:32:44.179 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-13 zones={test-zone3, test-zone1} 2021-03-12T02:32:44.229 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:32:44.282 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:32:44.349 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-13 zones={test-zone3, test-zone1} 2021-03-12T02:32:44.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:44.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:44.351 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:44.389 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:44.389 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:44.577 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.17.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:44.578 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-13 2021-03-12T02:32:44.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:44.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:44.580 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:44.624 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:44.880 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.17.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:44.881 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-13 2021-03-12T02:32:45.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:45.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:45.408 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:45.446 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:45.447 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:45.618 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.15.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:45.619 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-14 2021-03-12T02:32:45.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:45.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:45.621 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:45.665 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:45.666 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:45.873 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:45.874 WARNING:rgw_multi.tests:bucket bbucnh-14 zone test-zone1 behind zone test-zone2: shard=1 master=00000000002.15.5 target=00000000001.9.5 2021-03-12T02:32:50.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:50.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:50.877 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:50.920 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:50.920 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:51.119 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.15.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:51.119 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-14 2021-03-12T02:32:51.120 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-14 zones={test-zone1, test-zone2} 2021-03-12T02:32:52.062 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:32:52.070 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:32:52.179 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-14 zones={test-zone1, test-zone2} 2021-03-12T02:32:52.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:52.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:52.180 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:52.223 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:52.395 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.15.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:52.396 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-14 2021-03-12T02:32:52.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:52.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:52.397 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:52.442 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:52.443 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:52.655 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.15.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:52.656 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-14 2021-03-12T02:32:52.656 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-14 zones={test-zone3, test-zone2} 2021-03-12T02:32:52.972 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:32:53.027 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:32:53.088 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-14 zones={test-zone3, test-zone2} 2021-03-12T02:32:53.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:53.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:53.108 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:53.146 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:53.147 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:53.329 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.14.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:32:53.330 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-15 2021-03-12T02:32:53.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:53.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:53.331 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:53.414 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:53.414 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:53.624 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:53.625 WARNING:rgw_multi.tests:bucket bbucnh-15 zone test-zone1 behind zone test-zone3: shard=1 master=00000000002.14.5 target=00000000001.8.5 2021-03-12T02:32:58.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:58.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:32:58.627 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:32:58.672 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:32:58.673 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:32:58.907 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:32:58.908 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-15 2021-03-12T02:32:58.909 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-15 zones={test-zone1, test-zone3} 2021-03-12T02:33:00.518 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:33:00.527 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:33:00.654 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-15 zones={test-zone1, test-zone3} 2021-03-12T02:33:00.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:00.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:00.655 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:00.711 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:00.712 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:00.900 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.14.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:00.901 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-15 2021-03-12T02:33:00.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:00.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:00.902 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:00.947 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:00.948 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:01.129 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:01.129 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-15 2021-03-12T02:33:01.130 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-15 zones={test-zone2, test-zone3} 2021-03-12T02:33:02.281 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:33:02.288 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:33:02.387 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-15 zones={test-zone2, test-zone3} 2021-03-12T02:33:02.388 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2021-03-12T02:33:02.388 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-16 2021-03-12T02:33:02.709 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-17 2021-03-12T02:33:02.766 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-18 2021-03-12T02:33:02.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:02.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:02.867 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:02.903 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:02.903 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:03.074 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:33:03.104 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516382.866387_126.1","last_update":"2021-03-12T02:33:02.866387Z"},{"marker":"1_1615516309.597935_168.1","last_update":"2021-03-12T02:31:49.597935Z"}] 2021-03-12T02:33:03.104 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:33:03.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:03.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:03.106 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:03.147 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:03.147 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:03.322 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516336.084220_97.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:32:16.084220Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:33:03.323 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615516382.866387_126.1 target=1_1615516336.084220_97.1 2021-03-12T02:33:08.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:08.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:08.326 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:08.367 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:08.367 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:08.539 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516382.866387_126.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:33:02.866387Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:33:08.540 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:33:08.541 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:33:08.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:08.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:08.545 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:08.589 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:08.589 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:08.785 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516382.866387_126.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:33:02.866387Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:33:08.785 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:33:08.786 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:33:08.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:08.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:08.788 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:08.827 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:08.827 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:09.093 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516382.866387_126.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:33:02.866387Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:33:09.094 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:33:10.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:10.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:10.102 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:10.149 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:10.149 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:10.342 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.25.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:10.342 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:33:10.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:10.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:10.344 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:10.389 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:10.390 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:10.566 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:10.566 WARNING:rgw_multi.tests:bucket bbucnh-16 zone test-zone2 behind zone test-zone1: shard=5 master=00000000002.25.10 target= 2021-03-12T02:33:15.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:15.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:15.569 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:15.611 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:15.611 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:15.795 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5tu08427ssdgtBQhusV0YIQo6J8hJG4","ns":""},"count":1},"inc_marker":{"position":"00000000005.31.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:15.796 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:33:15.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:15.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:15.798 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:15.839 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:15.839 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:16.041 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:16.043 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:33:16.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:16.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:16.047 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:16.134 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:16.134 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:16.362 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5tu08427ssdgtBQhusV0YIQo6J8hJG4","ns":""},"count":1},"inc_marker":{"position":"00000000005.31.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:16.363 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:33:16.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:16.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:16.364 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:16.447 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:16.447 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:16.647 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:16.647 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:33:16.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:16.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:16.649 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:16.694 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:16.694 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:16.953 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.31.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:16.954 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:33:17.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:17.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:17.484 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:17.526 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:17.526 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:17.701 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.11.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:17.702 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-17 2021-03-12T02:33:17.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:17.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:17.703 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:17.747 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:17.747 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:17.976 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:17.977 WARNING:rgw_multi.tests:bucket bbucnh-17 zone test-zone1 behind zone test-zone2: shard=5 master=00000000002.11.10 target= 2021-03-12T02:33:22.982 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:22.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:22.984 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:23.027 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:23.027 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:23.247 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"h20YS3hno1mOfuJ3qS6hcx1pvmMyFcW","ns":""},"count":1},"inc_marker":{"position":"00000000005.17.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:23.248 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-17 2021-03-12T02:33:23.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:23.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:23.249 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:23.291 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:23.291 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:23.472 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.17.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:23.473 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-17 2021-03-12T02:33:23.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:23.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:23.475 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:23.518 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:23.518 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:23.744 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"h20YS3hno1mOfuJ3qS6hcx1pvmMyFcW","ns":""},"count":1},"inc_marker":{"position":"00000000005.17.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:23.745 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-17 2021-03-12T02:33:24.267 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:24.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:24.268 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:24.306 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:24.307 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:24.516 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.17.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:24.516 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-18 2021-03-12T02:33:24.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:24.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:24.518 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:24.601 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:24.601 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:24.836 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:24.837 WARNING:rgw_multi.tests:bucket bbucnh-18 zone test-zone1 behind zone test-zone3: shard=5 master=00000000002.17.10 target= 2021-03-12T02:33:29.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:29.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:29.840 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:29.884 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:29.884 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:30.121 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"pJq77Wp-iEF4-9cZIzmJNypsUK5SLQj","ns":""},"count":1},"inc_marker":{"position":"00000000006.25.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:30.122 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-18 2021-03-12T02:33:30.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:30.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:30.125 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:30.202 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:30.202 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:30.389 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.25.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:30.389 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-18 2021-03-12T02:33:30.390 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:30.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:30.391 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:30.435 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:30.435 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:30.631 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"pJq77Wp-iEF4-9cZIzmJNypsUK5SLQj","ns":""},"count":1},"inc_marker":{"position":"00000000006.25.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:30.632 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-18 2021-03-12T02:33:33.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:33.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:33.017 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:33.058 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:33.058 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:33.323 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.30.5"},{"key":2,"val":""},{"key":3,"val":"00000000002.20.10"},{"key":4,"val":""},{"key":5,"val":"00000000005.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:33.324 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:33:33.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:33.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:33.326 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:33.370 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:33.370 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:33.550 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"j68Khv5NHHxPvkvGV8Y05q7VX8rYD45","ns":""},"count":1},"inc_marker":{"position":"00000000010.30.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5tu08427ssdgtBQhusV0YIQo6J8hJG4","ns":""},"count":1},"inc_marker":{"position":"00000000005.31.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:33.550 WARNING:rgw_multi.tests:bucket bbucnh-16 zone test-zone2 behind zone test-zone1: shard=3 master=00000000002.20.10 target= 2021-03-12T02:33:38.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:38.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:38.553 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:38.598 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:38.599 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:38.789 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"j68Khv5NHHxPvkvGV8Y05q7VX8rYD45","ns":""},"count":1},"inc_marker":{"position":"00000000010.30.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UEZO4r9bNBGZ7AF0kQDvpOaNmtwaM6Y","ns":""},"count":1},"inc_marker":{"position":"00000000003.22.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5tu08427ssdgtBQhusV0YIQo6J8hJG4","ns":""},"count":1},"inc_marker":{"position":"00000000008.37.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:38.789 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:33:38.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:38.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:38.791 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:38.831 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:38.832 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:39.044 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.30.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.22.9"},{"key":4,"val":""},{"key":5,"val":"00000000008.37.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:39.045 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:33:39.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:39.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:39.046 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:39.133 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:39.335 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"j68Khv5NHHxPvkvGV8Y05q7VX8rYD45","ns":""},"count":1},"inc_marker":{"position":"00000000010.30.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UEZO4r9bNBGZ7AF0kQDvpOaNmtwaM6Y","ns":""},"count":1},"inc_marker":{"position":"00000000003.22.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5tu08427ssdgtBQhusV0YIQo6J8hJG4","ns":""},"count":1},"inc_marker":{"position":"00000000008.37.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:39.336 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:33:39.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:39.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:39.337 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:39.417 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:39.417 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:39.619 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.30.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.22.9"},{"key":4,"val":""},{"key":5,"val":"00000000008.37.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:39.619 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:33:39.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:39.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:39.621 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:39.666 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:39.666 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:39.930 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.30.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.22.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.37.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:39.931 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:33:39.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:39.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:39.933 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:39.973 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:39.974 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:40.152 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.16.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.33.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.30.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:40.152 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-16 2021-03-12T02:33:40.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:40.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:40.154 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:40.197 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:40.197 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:40.432 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"j68Khv5NHHxPvkvGV8Y05q7VX8rYD45","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UEZO4r9bNBGZ7AF0kQDvpOaNmtwaM6Y","ns":""},"count":1},"inc_marker":{"position":"00000000010.33.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5tu08427ssdgtBQhusV0YIQo6J8hJG4","ns":""},"count":1},"inc_marker":{"position":"00000000007.30.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:40.433 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-16 2021-03-12T02:33:40.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:40.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:40.434 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:40.476 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:40.477 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:40.674 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.16.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.33.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.30.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:40.675 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-16 2021-03-12T02:33:40.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:40.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:40.676 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:40.721 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:40.721 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:40.954 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"j68Khv5NHHxPvkvGV8Y05q7VX8rYD45","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UEZO4r9bNBGZ7AF0kQDvpOaNmtwaM6Y","ns":""},"count":1},"inc_marker":{"position":"00000000010.33.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5tu08427ssdgtBQhusV0YIQo6J8hJG4","ns":""},"count":1},"inc_marker":{"position":"00000000007.30.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:40.954 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-16 2021-03-12T02:33:40.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:40.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:40.956 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:41.039 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:41.040 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:41.237 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.18.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.17.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.46.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:41.238 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-16 2021-03-12T02:33:41.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:41.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:41.239 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:41.328 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:41.328 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:41.563 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"j68Khv5NHHxPvkvGV8Y05q7VX8rYD45","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UEZO4r9bNBGZ7AF0kQDvpOaNmtwaM6Y","ns":""},"count":1},"inc_marker":{"position":"00000000003.17.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5tu08427ssdgtBQhusV0YIQo6J8hJG4","ns":""},"count":1},"inc_marker":{"position":"00000000014.46.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:41.564 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-16 2021-03-12T02:33:41.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:41.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:41.566 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:41.647 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:41.746 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:41.852 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.18.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.17.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.46.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:41.852 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-16 2021-03-12T02:33:41.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:41.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:41.854 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:41.900 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:41.900 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:42.106 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"j68Khv5NHHxPvkvGV8Y05q7VX8rYD45","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UEZO4r9bNBGZ7AF0kQDvpOaNmtwaM6Y","ns":""},"count":1},"inc_marker":{"position":"00000000003.17.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5tu08427ssdgtBQhusV0YIQo6J8hJG4","ns":""},"count":1},"inc_marker":{"position":"00000000014.46.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:42.106 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-16 2021-03-12T02:33:42.107 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-16 zones={test-zone2, test-zone1} 2021-03-12T02:33:43.245 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:33:43.252 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:33:43.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:33:43.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:33:43.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:33:43.254 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:33:43.261 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:33:43.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:33:43.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:33:43.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:33:43.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:43.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:43.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:43.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:43.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:43.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:43.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:43.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:43.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:43.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:44.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:44.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:44.259 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-16 zones={test-zone2, test-zone1} 2021-03-12T02:33:44.259 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-16 zones={test-zone3, test-zone1} 2021-03-12T02:33:44.755 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:33:44.761 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:33:44.762 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:33:44.762 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:33:44.762 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:33:44.763 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:33:44.768 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:33:44.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:33:44.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:33:44.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:33:44.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:44.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:44.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:44.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:44.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:45.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:45.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:45.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:45.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:45.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:45.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:45.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:45.743 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-16 zones={test-zone3, test-zone1} 2021-03-12T02:33:45.743 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-16 zones={test-zone3, test-zone2} 2021-03-12T02:33:46.180 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:33:46.190 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:33:46.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:33:46.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:33:46.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:33:46.192 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:33:46.199 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:33:46.199 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:33:46.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:33:46.200 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:33:46.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:46.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:46.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:46.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:46.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:46.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:46.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:46.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:46.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:46.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:46.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:47.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:47.185 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-16 zones={test-zone3, test-zone2} 2021-03-12T02:33:47.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:47.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:47.187 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:47.230 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:47.231 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:47.435 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.32.5"},{"key":2,"val":""},{"key":3,"val":"00000000006.29.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.24.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:47.436 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-17 2021-03-12T02:33:47.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:47.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:47.437 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:47.483 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:47.483 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:47.683 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"GtTJ1CoQqFpb84tWlIgo2icYaplaRI7","ns":""},"count":1},"inc_marker":{"position":"00000000010.32.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4eREjbySR2Zd2B5gndRe4B2vfB9HlXl","ns":""},"count":1},"inc_marker":{"position":"00000000006.29.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"h20YS3hno1mOfuJ3qS6hcx1pvmMyFcW","ns":""},"count":1},"inc_marker":{"position":"00000000007.24.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:47.684 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-17 2021-03-12T02:33:47.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:47.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:47.686 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:47.728 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:47.728 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:47.949 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.32.5"},{"key":2,"val":""},{"key":3,"val":"00000000006.29.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.24.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:47.949 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-17 2021-03-12T02:33:47.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:47.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:47.951 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:48.032 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:48.033 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:48.255 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"GtTJ1CoQqFpb84tWlIgo2icYaplaRI7","ns":""},"count":1},"inc_marker":{"position":"00000000010.32.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4eREjbySR2Zd2B5gndRe4B2vfB9HlXl","ns":""},"count":1},"inc_marker":{"position":"00000000006.29.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"h20YS3hno1mOfuJ3qS6hcx1pvmMyFcW","ns":""},"count":1},"inc_marker":{"position":"00000000007.24.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:48.256 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-17 2021-03-12T02:33:48.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:48.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:48.257 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:48.337 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:48.337 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:48.545 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.32.5"},{"key":2,"val":""},{"key":3,"val":"00000000006.29.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.24.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:48.546 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-17 2021-03-12T02:33:48.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:48.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:48.548 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:48.588 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:48.589 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:48.850 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.32.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.29.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.24.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:48.850 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-17 2021-03-12T02:33:48.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:48.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:48.852 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:48.931 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:48.931 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:49.120 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.17.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.40.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:49.121 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-17 2021-03-12T02:33:49.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:49.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:49.122 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:49.166 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:49.167 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:49.368 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"GtTJ1CoQqFpb84tWlIgo2icYaplaRI7","ns":""},"count":1},"inc_marker":{"position":"00000000003.17.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"6XAmoNLrROyvWLj7xZaFNq4FswLRYh.","ns":""},"count":1},"inc_marker":{"position":"00000000010.40.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"h20YS3hno1mOfuJ3qS6hcx1pvmMyFcW","ns":""},"count":1},"inc_marker":{"position":"00000000008.23.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:49.368 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-17 2021-03-12T02:33:49.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:49.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:49.370 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:49.413 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:49.594 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.17.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.40.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:49.595 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-17 2021-03-12T02:33:49.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:49.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:49.597 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:49.639 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:49.640 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:49.856 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"GtTJ1CoQqFpb84tWlIgo2icYaplaRI7","ns":""},"count":1},"inc_marker":{"position":"00000000003.17.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"6XAmoNLrROyvWLj7xZaFNq4FswLRYh.","ns":""},"count":1},"inc_marker":{"position":"00000000010.40.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"h20YS3hno1mOfuJ3qS6hcx1pvmMyFcW","ns":""},"count":1},"inc_marker":{"position":"00000000008.23.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:49.856 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-17 2021-03-12T02:33:49.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:49.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:49.858 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:49.938 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:49.939 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:50.125 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.14.9"},{"key":2,"val":""},{"key":3,"val":"00000000006.32.5"},{"key":4,"val":""},{"key":5,"val":"00000000014.34.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:50.126 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-17 2021-03-12T02:33:50.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:50.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:50.128 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:50.212 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:50.212 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:50.426 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"GtTJ1CoQqFpb84tWlIgo2icYaplaRI7","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4eREjbySR2Zd2B5gndRe4B2vfB9HlXl","ns":""},"count":1},"inc_marker":{"position":"00000000006.32.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"h20YS3hno1mOfuJ3qS6hcx1pvmMyFcW","ns":""},"count":1},"inc_marker":{"position":"00000000014.34.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:50.426 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-17 2021-03-12T02:33:50.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:50.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:50.428 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:50.507 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:50.507 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:50.715 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.14.9"},{"key":2,"val":""},{"key":3,"val":"00000000006.32.5"},{"key":4,"val":""},{"key":5,"val":"00000000014.34.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:50.715 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-17 2021-03-12T02:33:50.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:50.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:50.717 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:50.760 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:50.760 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:50.983 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"GtTJ1CoQqFpb84tWlIgo2icYaplaRI7","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4eREjbySR2Zd2B5gndRe4B2vfB9HlXl","ns":""},"count":1},"inc_marker":{"position":"00000000006.32.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"h20YS3hno1mOfuJ3qS6hcx1pvmMyFcW","ns":""},"count":1},"inc_marker":{"position":"00000000014.34.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:50.984 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-17 2021-03-12T02:33:50.985 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-17 zones={test-zone2, test-zone1} 2021-03-12T02:33:52.419 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:33:52.425 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:33:52.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:33:52.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:33:52.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:33:52.427 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:33:52.434 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:33:52.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:33:52.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:33:52.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:33:52.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:52.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:52.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:52.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:52.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:52.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:52.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:52.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:53.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:53.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:53.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:53.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:53.398 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-17 zones={test-zone2, test-zone1} 2021-03-12T02:33:53.399 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-17 zones={test-zone3, test-zone1} 2021-03-12T02:33:53.885 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:33:53.891 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:33:53.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:33:53.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:33:53.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:33:53.893 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:33:53.898 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:33:53.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:33:53.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:33:53.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:33:53.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:53.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:53.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:54.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:54.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:54.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:54.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:54.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:54.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:54.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:54.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:54.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:54.926 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-17 zones={test-zone3, test-zone1} 2021-03-12T02:33:54.927 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-17 zones={test-zone3, test-zone2} 2021-03-12T02:33:55.668 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:33:55.676 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:33:55.676 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:33:55.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:33:55.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:33:55.678 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:33:55.683 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:33:55.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:33:55.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:33:55.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:33:55.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:55.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:55.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:33:55.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:55.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:55.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:33:56.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:56.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:56.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:33:56.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:56.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:56.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:33:56.664 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-17 zones={test-zone3, test-zone2} 2021-03-12T02:33:56.664 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:56.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:56.666 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:56.708 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:56.708 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:56.910 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.29.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.20.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:56.911 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-18 2021-03-12T02:33:56.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:56.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:56.913 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:56.958 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:56.960 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:57.178 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7MacFH52SUYrqI-EjjbSHGgjmTj1Adn","ns":""},"count":1},"inc_marker":{"position":"00000000010.29.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"kp8brTrLkuFqEFRngYc2R8eAH8C2nz-","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"pJq77Wp-iEF4-9cZIzmJNypsUK5SLQj","ns":""},"count":1},"inc_marker":{"position":"00000000007.31.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:57.178 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-18 2021-03-12T02:33:57.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:57.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:57.180 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:57.219 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:57.220 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:57.415 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.29.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.20.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:57.416 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-18 2021-03-12T02:33:57.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:57.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:57.418 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:57.504 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:57.504 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:57.728 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"S9PKHWsHd58FBb4NTWZMTHnLgKB.mHA","ns":""},"count":1},"inc_marker":{"position":"00000000010.29.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"kp8brTrLkuFqEFRngYc2R8eAH8C2nz-","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"pJq77Wp-iEF4-9cZIzmJNypsUK5SLQj","ns":""},"count":1},"inc_marker":{"position":"00000000007.31.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:57.728 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-18 2021-03-12T02:33:57.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:57.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:57.730 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:57.812 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:57.813 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:58.007 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.29.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.20.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:58.007 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-18 2021-03-12T02:33:58.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:58.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:58.009 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:58.052 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:58.053 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:58.317 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.29.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.20.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.31.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:58.318 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-18 2021-03-12T02:33:58.318 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:58.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:58.319 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:58.402 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:58.403 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:58.574 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.13.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.36.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.26.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:58.574 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-18 2021-03-12T02:33:58.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:58.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:58.576 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:58.619 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:58.619 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:58.824 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7MacFH52SUYrqI-EjjbSHGgjmTj1Adn","ns":""},"count":1},"inc_marker":{"position":"00000000003.13.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"kp8brTrLkuFqEFRngYc2R8eAH8C2nz-","ns":""},"count":1},"inc_marker":{"position":"00000000010.36.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"pJq77Wp-iEF4-9cZIzmJNypsUK5SLQj","ns":""},"count":1},"inc_marker":{"position":"00000000007.26.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:58.824 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-18 2021-03-12T02:33:58.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:58.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:58.826 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:58.866 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:58.866 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:59.055 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.13.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.36.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.26.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:59.056 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-18 2021-03-12T02:33:59.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:59.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:59.058 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:59.104 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:59.104 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:59.325 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7MacFH52SUYrqI-EjjbSHGgjmTj1Adn","ns":""},"count":1},"inc_marker":{"position":"00000000003.13.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"kp8brTrLkuFqEFRngYc2R8eAH8C2nz-","ns":""},"count":1},"inc_marker":{"position":"00000000010.36.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"pJq77Wp-iEF4-9cZIzmJNypsUK5SLQj","ns":""},"count":1},"inc_marker":{"position":"00000000007.26.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:59.325 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-18 2021-03-12T02:33:59.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:59.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:59.327 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:59.366 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:59.367 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:59.571 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.14.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.22.9"},{"key":4,"val":""},{"key":5,"val":"00000000016.45.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:33:59.572 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-18 2021-03-12T02:33:59.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:59.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:59.573 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:59.658 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:59.658 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:33:59.870 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7MacFH52SUYrqI-EjjbSHGgjmTj1Adn","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"kp8brTrLkuFqEFRngYc2R8eAH8C2nz-","ns":""},"count":1},"inc_marker":{"position":"00000000003.22.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"pJq77Wp-iEF4-9cZIzmJNypsUK5SLQj","ns":""},"count":1},"inc_marker":{"position":"00000000016.45.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:33:59.870 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-18 2021-03-12T02:33:59.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:59.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:33:59.872 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:33:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:33:59.951 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:00.152 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.14.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.22.9"},{"key":4,"val":""},{"key":5,"val":"00000000016.45.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:00.153 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-18 2021-03-12T02:34:00.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:00.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:00.155 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:00.196 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:00.197 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:00.426 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7MacFH52SUYrqI-EjjbSHGgjmTj1Adn","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"kp8brTrLkuFqEFRngYc2R8eAH8C2nz-","ns":""},"count":1},"inc_marker":{"position":"00000000003.22.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"pJq77Wp-iEF4-9cZIzmJNypsUK5SLQj","ns":""},"count":1},"inc_marker":{"position":"00000000016.45.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:00.427 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-18 2021-03-12T02:34:00.427 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-18 zones={test-zone2, test-zone1} 2021-03-12T02:34:01.126 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:34:01.133 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:01.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:01.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:01.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:01.135 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:34:01.142 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:01.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:01.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:01.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:01.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:01.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:01.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:01.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:01.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:01.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:01.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:01.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:01.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:01.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:01.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:02.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:02.119 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-18 zones={test-zone2, test-zone1} 2021-03-12T02:34:02.120 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-18 zones={test-zone3, test-zone1} 2021-03-12T02:34:02.723 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:34:02.730 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:02.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:02.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:02.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:02.732 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:34:02.737 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:02.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:02.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:02.739 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:02.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:02.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:02.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:02.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:02.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:03.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:03.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:03.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:03.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:03.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:03.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:03.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:03.727 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-18 zones={test-zone3, test-zone1} 2021-03-12T02:34:03.728 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-18 zones={test-zone3, test-zone2} 2021-03-12T02:34:04.096 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:34:04.105 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:04.105 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:04.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:04.106 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:04.106 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:34:04.114 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:04.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:04.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:04.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:04.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:04.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:04.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:04.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:04.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:04.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:04.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:04.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:04.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:04.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:04.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:05.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:05.113 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-18 zones={test-zone3, test-zone2} 2021-03-12T02:34:06.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:06.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:06.979 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:07.021 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:07.022 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:07.204 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.36.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.22.9"},{"key":4,"val":""},{"key":5,"val":"00000000008.37.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:07.205 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:34:07.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:07.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:07.207 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:07.252 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:07.253 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:07.448 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"j68Khv5NHHxPvkvGV8Y05q7VX8rYD45","ns":""},"count":1},"inc_marker":{"position":"00000000011.36.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UEZO4r9bNBGZ7AF0kQDvpOaNmtwaM6Y","ns":""},"count":1},"inc_marker":{"position":"00000000003.22.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5tu08427ssdgtBQhusV0YIQo6J8hJG4","ns":""},"count":1},"inc_marker":{"position":"00000000008.37.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:07.449 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:34:07.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:07.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:07.450 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:07.493 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:07.493 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:07.691 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.36.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.25.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.37.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:07.691 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:34:07.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:07.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:07.693 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:07.774 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:07.774 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:07.975 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"j68Khv5NHHxPvkvGV8Y05q7VX8rYD45","ns":""},"count":1},"inc_marker":{"position":"00000000011.36.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UEZO4r9bNBGZ7AF0kQDvpOaNmtwaM6Y","ns":""},"count":1},"inc_marker":{"position":"00000000003.22.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5tu08427ssdgtBQhusV0YIQo6J8hJG4","ns":""},"count":1},"inc_marker":{"position":"00000000008.37.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:07.976 WARNING:rgw_multi.tests:bucket bbucnh-16 zone test-zone3 behind zone test-zone1: shard=3 master=00000000004.25.5 target=00000000003.22.9 2021-03-12T02:34:12.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:12.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:12.979 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:13.022 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:13.023 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:13.236 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"j68Khv5NHHxPvkvGV8Y05q7VX8rYD45","ns":""},"count":1},"inc_marker":{"position":"00000000011.36.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UEZO4r9bNBGZ7AF0kQDvpOaNmtwaM6Y","ns":""},"count":1},"inc_marker":{"position":"00000000004.25.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5tu08427ssdgtBQhusV0YIQo6J8hJG4","ns":""},"count":1},"inc_marker":{"position":"00000000009.40.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:13.236 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:34:13.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:13.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:13.238 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:13.320 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:13.320 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:13.514 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.36.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.25.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.40.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:13.515 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:34:13.515 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:13.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:13.516 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:13.560 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:13.561 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:13.822 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.36.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.25.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000009.40.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:13.823 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-16 2021-03-12T02:34:13.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:13.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:13.824 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:13.866 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:13.866 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:14.082 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.19.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.39.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.33.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:14.083 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-16 2021-03-12T02:34:14.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:14.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:14.085 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:14.129 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:14.130 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:14.320 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"j68Khv5NHHxPvkvGV8Y05q7VX8rYD45","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UEZO4r9bNBGZ7AF0kQDvpOaNmtwaM6Y","ns":""},"count":1},"inc_marker":{"position":"00000000011.39.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5tu08427ssdgtBQhusV0YIQo6J8hJG4","ns":""},"count":1},"inc_marker":{"position":"00000000008.33.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:14.321 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-16 2021-03-12T02:34:14.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:14.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:14.322 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:14.364 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:14.365 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:14.552 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.19.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.39.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.33.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:14.552 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-16 2021-03-12T02:34:14.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:14.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:14.554 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:14.598 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:14.599 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:14.810 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"j68Khv5NHHxPvkvGV8Y05q7VX8rYD45","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UEZO4r9bNBGZ7AF0kQDvpOaNmtwaM6Y","ns":""},"count":1},"inc_marker":{"position":"00000000011.39.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5tu08427ssdgtBQhusV0YIQo6J8hJG4","ns":""},"count":1},"inc_marker":{"position":"00000000008.33.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:14.810 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-16 2021-03-12T02:34:14.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:14.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:14.812 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:14.894 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:14.895 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:15.087 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.21.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.20.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.49.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:15.088 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-16 2021-03-12T02:34:15.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:15.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:15.090 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:15.170 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:15.402 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"j68Khv5NHHxPvkvGV8Y05q7VX8rYD45","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UEZO4r9bNBGZ7AF0kQDvpOaNmtwaM6Y","ns":""},"count":1},"inc_marker":{"position":"00000000004.20.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5tu08427ssdgtBQhusV0YIQo6J8hJG4","ns":""},"count":1},"inc_marker":{"position":"00000000015.49.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:15.403 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-16 2021-03-12T02:34:15.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:15.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:15.405 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:15.476 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:15.476 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:15.675 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.21.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.20.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.49.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:15.675 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-16 2021-03-12T02:34:15.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:15.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:15.677 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:15.722 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:15.723 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:15.941 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"j68Khv5NHHxPvkvGV8Y05q7VX8rYD45","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UEZO4r9bNBGZ7AF0kQDvpOaNmtwaM6Y","ns":""},"count":1},"inc_marker":{"position":"00000000004.20.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"5tu08427ssdgtBQhusV0YIQo6J8hJG4","ns":""},"count":1},"inc_marker":{"position":"00000000015.49.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:15.942 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-16 2021-03-12T02:34:15.942 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-16 zones={test-zone2, test-zone1} 2021-03-12T02:34:16.916 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:34:16.923 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:16.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:16.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:16.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:16.925 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:34:16.934 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:16.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:16.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:16.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:16.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:16.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:17.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:17.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:17.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:17.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:17.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:17.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:17.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:17.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:17.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:17.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:17.953 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-16 zones={test-zone2, test-zone1} 2021-03-12T02:34:17.953 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-16 zones={test-zone3, test-zone1} 2021-03-12T02:34:18.838 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:34:18.844 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:18.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:18.845 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:18.846 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:18.846 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:34:18.853 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:18.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:18.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:18.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:18.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:18.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:18.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:19.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:19.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:19.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:19.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:19.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:19.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:19.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:19.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:19.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:19.826 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-16 zones={test-zone3, test-zone1} 2021-03-12T02:34:19.827 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-16 zones={test-zone3, test-zone2} 2021-03-12T02:34:19.857 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:34:19.865 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:19.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:19.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:19.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:19.867 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:34:19.874 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:19.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:19.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:19.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:19.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:19.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:19.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:20.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:20.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:20.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:20.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:20.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:20.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:20.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:20.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:20.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:20.872 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-16 zones={test-zone3, test-zone2} 2021-03-12T02:34:20.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:20.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:20.874 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:20.915 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:20.915 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:21.105 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.38.5"},{"key":2,"val":""},{"key":3,"val":"00000000007.33.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.27.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:21.105 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-17 2021-03-12T02:34:21.106 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:21.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:21.107 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:21.150 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:21.150 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:21.362 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"GtTJ1CoQqFpb84tWlIgo2icYaplaRI7","ns":""},"count":1},"inc_marker":{"position":"00000000011.38.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4eREjbySR2Zd2B5gndRe4B2vfB9HlXl","ns":""},"count":1},"inc_marker":{"position":"00000000007.33.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"h20YS3hno1mOfuJ3qS6hcx1pvmMyFcW","ns":""},"count":1},"inc_marker":{"position":"00000000008.27.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:21.363 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-17 2021-03-12T02:34:21.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:21.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:21.364 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:21.408 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:21.408 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:21.611 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.38.5"},{"key":2,"val":""},{"key":3,"val":"00000000007.33.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.27.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:21.611 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-17 2021-03-12T02:34:21.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:21.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:21.613 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:21.699 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:21.699 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:21.898 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"GtTJ1CoQqFpb84tWlIgo2icYaplaRI7","ns":""},"count":1},"inc_marker":{"position":"00000000011.38.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4eREjbySR2Zd2B5gndRe4B2vfB9HlXl","ns":""},"count":1},"inc_marker":{"position":"00000000007.33.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"h20YS3hno1mOfuJ3qS6hcx1pvmMyFcW","ns":""},"count":1},"inc_marker":{"position":"00000000008.27.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:21.899 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-17 2021-03-12T02:34:21.899 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:21.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:21.900 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:21.986 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:21.986 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:22.181 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.38.5"},{"key":2,"val":""},{"key":3,"val":"00000000007.33.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.27.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:22.181 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-17 2021-03-12T02:34:22.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:22.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:22.183 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:22.227 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:22.228 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:22.486 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.38.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.33.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.27.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:22.487 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-17 2021-03-12T02:34:22.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:22.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:22.489 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:22.574 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:22.575 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:22.768 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.20.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.45.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.26.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:22.769 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-17 2021-03-12T02:34:22.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:22.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:22.771 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:22.815 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:22.816 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:23.021 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"GtTJ1CoQqFpb84tWlIgo2icYaplaRI7","ns":""},"count":1},"inc_marker":{"position":"00000000004.20.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"6XAmoNLrROyvWLj7xZaFNq4FswLRYh.","ns":""},"count":1},"inc_marker":{"position":"00000000011.45.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"h20YS3hno1mOfuJ3qS6hcx1pvmMyFcW","ns":""},"count":1},"inc_marker":{"position":"00000000009.26.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:23.022 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-17 2021-03-12T02:34:23.022 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:23.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:23.023 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:23.063 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:23.064 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:23.270 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.20.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.45.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.26.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:23.270 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-17 2021-03-12T02:34:23.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:23.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:23.272 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:23.317 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:23.317 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:23.526 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"GtTJ1CoQqFpb84tWlIgo2icYaplaRI7","ns":""},"count":1},"inc_marker":{"position":"00000000004.20.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"6XAmoNLrROyvWLj7xZaFNq4FswLRYh.","ns":""},"count":1},"inc_marker":{"position":"00000000011.45.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"h20YS3hno1mOfuJ3qS6hcx1pvmMyFcW","ns":""},"count":1},"inc_marker":{"position":"00000000009.26.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:23.526 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-17 2021-03-12T02:34:23.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:23.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:23.528 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:23.612 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:23.612 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:23.803 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.17.5"},{"key":2,"val":""},{"key":3,"val":"00000000007.36.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.37.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:23.803 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-17 2021-03-12T02:34:23.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:23.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:23.805 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:23.887 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:23.887 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:24.097 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"GtTJ1CoQqFpb84tWlIgo2icYaplaRI7","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4eREjbySR2Zd2B5gndRe4B2vfB9HlXl","ns":""},"count":1},"inc_marker":{"position":"00000000007.36.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"h20YS3hno1mOfuJ3qS6hcx1pvmMyFcW","ns":""},"count":1},"inc_marker":{"position":"00000000015.37.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:24.098 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-17 2021-03-12T02:34:24.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:24.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:24.099 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:24.182 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:24.182 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:24.368 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.17.5"},{"key":2,"val":""},{"key":3,"val":"00000000007.36.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.37.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:24.369 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-17 2021-03-12T02:34:24.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:24.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:24.370 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:24.414 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:24.637 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"GtTJ1CoQqFpb84tWlIgo2icYaplaRI7","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"4eREjbySR2Zd2B5gndRe4B2vfB9HlXl","ns":""},"count":1},"inc_marker":{"position":"00000000007.36.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"h20YS3hno1mOfuJ3qS6hcx1pvmMyFcW","ns":""},"count":1},"inc_marker":{"position":"00000000015.37.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:24.637 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-17 2021-03-12T02:34:24.638 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-17 zones={test-zone2, test-zone1} 2021-03-12T02:34:24.894 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:34:24.900 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:24.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:24.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:24.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:24.902 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:34:24.909 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:24.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:24.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:24.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:24.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:24.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:25.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:25.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:25.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:25.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:25.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:25.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:25.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:25.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:25.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:25.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:25.898 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-17 zones={test-zone2, test-zone1} 2021-03-12T02:34:25.899 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-17 zones={test-zone3, test-zone1} 2021-03-12T02:34:26.354 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:34:26.360 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:26.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:26.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:26.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:26.361 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:34:26.366 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:26.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:26.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:26.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:26.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:26.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:26.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:26.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:26.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:26.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:26.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:26.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:26.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:27.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:27.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:27.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:27.338 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-17 zones={test-zone3, test-zone1} 2021-03-12T02:34:27.338 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-17 zones={test-zone3, test-zone2} 2021-03-12T02:34:28.156 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:34:28.165 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:28.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:28.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:28.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:28.167 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:34:28.174 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:28.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:28.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:28.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:28.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:28.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:28.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:28.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:28.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:28.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:28.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:28.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:28.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:28.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:28.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:29.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:29.205 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-17 zones={test-zone3, test-zone2} 2021-03-12T02:34:29.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:29.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:29.207 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:29.250 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:29.251 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:29.423 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.35.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.23.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.34.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:29.423 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-18 2021-03-12T02:34:29.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:29.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:29.425 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:29.471 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:29.472 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:29.678 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7MacFH52SUYrqI-EjjbSHGgjmTj1Adn","ns":""},"count":1},"inc_marker":{"position":"00000000011.35.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"kp8brTrLkuFqEFRngYc2R8eAH8C2nz-","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"pJq77Wp-iEF4-9cZIzmJNypsUK5SLQj","ns":""},"count":1},"inc_marker":{"position":"00000000008.34.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:29.678 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-18 2021-03-12T02:34:29.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:29.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:29.680 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:29.720 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:29.720 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:29.916 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.35.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.23.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.34.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:29.917 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-18 2021-03-12T02:34:29.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:29.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:29.918 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:30.003 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:30.004 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:30.215 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"S9PKHWsHd58FBb4NTWZMTHnLgKB.mHA","ns":""},"count":1},"inc_marker":{"position":"00000000011.35.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"kp8brTrLkuFqEFRngYc2R8eAH8C2nz-","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"pJq77Wp-iEF4-9cZIzmJNypsUK5SLQj","ns":""},"count":1},"inc_marker":{"position":"00000000008.34.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:30.216 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-18 2021-03-12T02:34:30.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:30.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:30.217 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:30.264 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:30.453 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.35.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.23.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.34.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:30.453 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-18 2021-03-12T02:34:30.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:30.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:30.455 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:30.501 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:30.502 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:30.758 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.35.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.23.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.34.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:30.759 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-18 2021-03-12T02:34:30.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:30.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:30.761 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:30.799 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:30.799 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:30.991 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.16.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.42.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.29.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:30.991 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-18 2021-03-12T02:34:30.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:30.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:30.993 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:31.035 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:31.036 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:31.253 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7MacFH52SUYrqI-EjjbSHGgjmTj1Adn","ns":""},"count":1},"inc_marker":{"position":"00000000004.16.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"kp8brTrLkuFqEFRngYc2R8eAH8C2nz-","ns":""},"count":1},"inc_marker":{"position":"00000000011.42.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"pJq77Wp-iEF4-9cZIzmJNypsUK5SLQj","ns":""},"count":1},"inc_marker":{"position":"00000000008.29.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:31.253 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-18 2021-03-12T02:34:31.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:31.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:31.255 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:31.295 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:31.295 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:31.484 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.16.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.42.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.29.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:31.484 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-18 2021-03-12T02:34:31.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:31.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:31.486 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:31.530 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:31.531 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:31.729 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7MacFH52SUYrqI-EjjbSHGgjmTj1Adn","ns":""},"count":1},"inc_marker":{"position":"00000000004.16.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"kp8brTrLkuFqEFRngYc2R8eAH8C2nz-","ns":""},"count":1},"inc_marker":{"position":"00000000011.42.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"pJq77Wp-iEF4-9cZIzmJNypsUK5SLQj","ns":""},"count":1},"inc_marker":{"position":"00000000008.29.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:31.730 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-18 2021-03-12T02:34:31.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:31.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:31.731 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:31.812 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:31.813 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:32.006 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.17.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.25.5"},{"key":4,"val":""},{"key":5,"val":"00000000017.48.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:32.007 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-18 2021-03-12T02:34:32.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:32.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:32.008 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:32.094 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:32.094 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:32.296 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7MacFH52SUYrqI-EjjbSHGgjmTj1Adn","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"kp8brTrLkuFqEFRngYc2R8eAH8C2nz-","ns":""},"count":1},"inc_marker":{"position":"00000000004.25.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"pJq77Wp-iEF4-9cZIzmJNypsUK5SLQj","ns":""},"count":1},"inc_marker":{"position":"00000000017.48.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:32.297 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-18 2021-03-12T02:34:32.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:32.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:32.299 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:32.380 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:32.380 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:32.580 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.17.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.25.5"},{"key":4,"val":""},{"key":5,"val":"00000000017.48.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:32.581 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-18 2021-03-12T02:34:32.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:32.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:32.583 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:32.626 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:32.626 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:32.828 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"7MacFH52SUYrqI-EjjbSHGgjmTj1Adn","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"kp8brTrLkuFqEFRngYc2R8eAH8C2nz-","ns":""},"count":1},"inc_marker":{"position":"00000000004.25.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"pJq77Wp-iEF4-9cZIzmJNypsUK5SLQj","ns":""},"count":1},"inc_marker":{"position":"00000000017.48.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:32.829 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-18 2021-03-12T02:34:32.829 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-18 zones={test-zone2, test-zone1} 2021-03-12T02:34:33.580 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:34:33.587 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:33.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:33.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:33.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:33.589 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:34:33.597 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:33.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:33.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:33.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:33.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:33.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:33.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:33.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:33.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:33.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:34.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:34.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:34.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:34.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:34.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:34.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:34.586 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-18 zones={test-zone2, test-zone1} 2021-03-12T02:34:34.587 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-18 zones={test-zone3, test-zone1} 2021-03-12T02:34:34.620 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:34:34.625 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:34.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:34.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:34.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:34.628 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:34:34.635 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:34.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:34.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:34.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:34.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:34.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:34.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:34.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:34.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:34.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:35.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:35.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:35.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:35.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:35.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:35.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:35.610 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-18 zones={test-zone3, test-zone1} 2021-03-12T02:34:35.611 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-18 zones={test-zone3, test-zone2} 2021-03-12T02:34:35.677 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:34:35.686 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:35.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:35.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:35.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:35.689 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:34:35.700 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:35.701 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-12T02:34:35.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-12T02:34:35.703 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-12T02:34:35.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:35.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:35.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:35.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:35.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:36.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-12T02:34:36.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:36.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:36.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-12T02:34:36.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:36.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:36.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-12T02:34:36.685 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-18 zones={test-zone3, test-zone2} 2021-03-12T02:34:36.686 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2021-03-12T02:34:37.647 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:37.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:37.648 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:37.686 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:37.687 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:37.857 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:34:37.885 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516477.646256_184.1","last_update":"2021-03-12T02:34:37.646256Z"},{"marker":"1_1615516309.597935_168.1","last_update":"2021-03-12T02:31:49.597935Z"}] 2021-03-12T02:34:37.886 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:34:37.886 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:37.887 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:37.887 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:37.927 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:37.928 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:38.128 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516382.866387_126.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:33:02.866387Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:34:38.129 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615516477.646256_184.1 target=1_1615516382.866387_126.1 2021-03-12T02:34:43.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:44.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:44.128 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:44.171 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:44.171 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:44.365 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516477.646256_184.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:37.646256Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:34:44.365 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:34:44.366 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:34:44.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:44.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:44.367 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:44.408 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:44.409 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:44.607 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516477.646256_184.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:37.646256Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:34:44.607 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:34:44.608 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:34:44.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:44.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:44.609 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:44.651 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:44.651 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:44.912 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516477.646256_184.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:37.646256Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:34:44.913 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:34:45.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:45.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:45.572 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:45.614 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:45.614 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:45.808 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.24.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:45.808 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-19 2021-03-12T02:34:45.809 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:45.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:45.810 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:45.852 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:45.853 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:46.052 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:46.053 WARNING:rgw_multi.tests:bucket bbucnh-19 zone test-zone2 behind zone test-zone1: shard=5 master=00000000002.24.10 target= 2021-03-12T02:34:51.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:51.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:51.057 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:51.101 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:51.101 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:51.300 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"nxN8ja.fVJqnclTdr6Jt9QaYVGPtLh2","ns":""},"count":1},"inc_marker":{"position":"00000000005.30.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:51.301 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-19 2021-03-12T02:34:51.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:51.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:51.302 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:51.342 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:51.342 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:51.536 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.30.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:51.536 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-19 2021-03-12T02:34:51.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:51.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:51.538 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:51.622 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:51.622 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:51.828 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"nxN8ja.fVJqnclTdr6Jt9QaYVGPtLh2","ns":""},"count":1},"inc_marker":{"position":"00000000005.30.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:51.828 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-19 2021-03-12T02:34:51.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:51.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:51.830 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:51.909 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:51.909 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:52.103 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.30.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:52.104 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-19 2021-03-12T02:34:52.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:52.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:52.105 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:52.149 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:52.150 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:52.413 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.30.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:52.414 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-19 2021-03-12T02:34:52.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:52.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:52.415 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:52.454 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:52.454 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:52.641 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.32.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:52.642 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-19 2021-03-12T02:34:52.643 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:52.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:52.644 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:52.689 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:52.689 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:52.904 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"nxN8ja.fVJqnclTdr6Jt9QaYVGPtLh2","ns":""},"count":1},"inc_marker":{"position":"00000000004.32.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:52.905 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-19 2021-03-12T02:34:52.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:52.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:52.906 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:52.947 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:52.948 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:53.151 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.32.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:53.152 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-19 2021-03-12T02:34:53.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:53.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:53.155 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:53.203 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:53.204 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:53.398 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"nxN8ja.fVJqnclTdr6Jt9QaYVGPtLh2","ns":""},"count":1},"inc_marker":{"position":"00000000004.32.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:53.399 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-19 2021-03-12T02:34:53.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:53.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:53.401 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:53.438 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:53.439 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:53.632 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:53.633 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-19 2021-03-12T02:34:53.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:53.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:53.634 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:53.718 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:53.718 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:53.923 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"nxN8ja.fVJqnclTdr6Jt9QaYVGPtLh2","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:53.923 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-19 2021-03-12T02:34:53.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:53.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:53.925 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:54.006 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:54.006 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:54.184 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:54.184 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-19 2021-03-12T02:34:54.185 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:54.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:54.186 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:54.231 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:54.232 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:54.451 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"nxN8ja.fVJqnclTdr6Jt9QaYVGPtLh2","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:54.451 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-19 2021-03-12T02:34:54.452 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-19 zones={test-zone2, test-zone1} 2021-03-12T02:34:55.199 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:34:55.205 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:55.206 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:34:55.255 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:55.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:55.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:55.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:55.463 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-19 zones={test-zone2, test-zone1} 2021-03-12T02:34:55.464 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-19 zones={test-zone3, test-zone1} 2021-03-12T02:34:56.167 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:34:56.217 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:56.225 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:34:56.232 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:56.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:56.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:56.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:56.430 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-19 zones={test-zone3, test-zone1} 2021-03-12T02:34:56.431 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-19 zones={test-zone3, test-zone2} 2021-03-12T02:34:56.887 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:34:56.896 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:56.896 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:34:56.943 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:34:56.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:57.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:57.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:34:57.136 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-19 zones={test-zone3, test-zone2} 2021-03-12T02:34:58.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:58.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:58.992 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:59.033 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:59.033 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:59.218 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.30.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000042.133.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:59.219 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-19 2021-03-12T02:34:59.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:59.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:59.220 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:59.267 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:59.267 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:59.497 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"nxN8ja.fVJqnclTdr6Jt9QaYVGPtLh2","ns":""},"count":1},"inc_marker":{"position":"00000000005.30.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zvYI6HNHDoifR.xeFx4-6iidIBnFk3G","ns":""},"count":5},"inc_marker":{"position":"00000000044.138.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:34:59.498 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-19 2021-03-12T02:34:59.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:59.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:59.499 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:59.545 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:59.546 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:34:59.745 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.30.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000047.145.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:34:59.745 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-19 2021-03-12T02:34:59.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:59.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:34:59.747 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:34:59.830 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:34:59.831 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:00.064 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"nxN8ja.fVJqnclTdr6Jt9QaYVGPtLh2","ns":""},"count":1},"inc_marker":{"position":"00000000005.30.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zvYI6HNHDoifR.xeFx4-6iidIBnFk3G","ns":""},"count":5},"inc_marker":{"position":"00000000044.138.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:35:00.065 WARNING:rgw_multi.tests:bucket bbucnh-19 zone test-zone3 behind zone test-zone1: shard=8 master=00000000047.145.9 target=00000000044.138.11 2021-03-12T02:35:05.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:05.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:05.068 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:05.112 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:05.112 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:05.350 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"nxN8ja.fVJqnclTdr6Jt9QaYVGPtLh2","ns":""},"count":1},"inc_marker":{"position":"00000000005.30.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zvYI6HNHDoifR.xeFx4-6iidIBnFk3G","ns":""},"count":5},"inc_marker":{"position":"00000000047.145.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:35:05.350 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-19 2021-03-12T02:35:05.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:05.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:05.352 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:05.432 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:05.432 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:05.647 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.30.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000047.145.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:35:05.647 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-19 2021-03-12T02:35:05.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:05.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:05.649 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:05.694 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:05.695 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:05.955 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.30.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000047.145.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:35:05.955 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-19 2021-03-12T02:35:05.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:05.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:05.957 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:06.038 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:06.038 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:06.237 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.32.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000046.134.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:35:06.238 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-19 2021-03-12T02:35:06.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:06.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:06.240 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:06.284 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:06.284 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:06.508 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"nxN8ja.fVJqnclTdr6Jt9QaYVGPtLh2","ns":""},"count":1},"inc_marker":{"position":"00000000004.32.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zvYI6HNHDoifR.xeFx4-6iidIBnFk3G","ns":""},"count":13},"inc_marker":{"position":"00000000046.134.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:35:06.509 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-19 2021-03-12T02:35:06.509 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:06.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:06.510 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:06.551 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:06.748 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.32.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000046.134.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:35:06.748 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-19 2021-03-12T02:35:06.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:06.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:06.750 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:06.792 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:06.793 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:07.030 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"nxN8ja.fVJqnclTdr6Jt9QaYVGPtLh2","ns":""},"count":1},"inc_marker":{"position":"00000000004.32.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zvYI6HNHDoifR.xeFx4-6iidIBnFk3G","ns":""},"count":13},"inc_marker":{"position":"00000000046.134.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:35:07.031 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-19 2021-03-12T02:35:07.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:07.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:07.033 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:07.115 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:07.116 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:07.320 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000048.153.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:35:07.321 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-19 2021-03-12T02:35:07.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:07.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:07.323 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:07.405 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:07.405 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:07.639 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"nxN8ja.fVJqnclTdr6Jt9QaYVGPtLh2","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"kPhz1SQvCNX2PWFkMnivBxINSce3yy3","ns":""},"count":4},"inc_marker":{"position":"00000000048.153.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:35:07.639 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-19 2021-03-12T02:35:07.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:07.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:07.641 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:07.728 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:07.728 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:07.922 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000048.153.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:35:07.923 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-19 2021-03-12T02:35:07.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:07.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:07.925 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:07.973 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:07.973 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:08.191 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"nxN8ja.fVJqnclTdr6Jt9QaYVGPtLh2","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"kPhz1SQvCNX2PWFkMnivBxINSce3yy3","ns":""},"count":4},"inc_marker":{"position":"00000000048.153.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:35:08.192 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-19 2021-03-12T02:35:08.193 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-19 zones={test-zone2, test-zone1} 2021-03-12T02:35:09.261 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:35:10.959 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:35:10.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.966 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:10.972 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:35:11.173 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:35:11.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.186 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.186 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.187 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:11.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:35:11.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:35:11.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:35:11.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:11.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:11.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:11.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:11.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:11.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:11.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:11.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:12.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:12.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:12.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:12.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:12.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:12.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:12.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:12.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:12.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:12.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:13.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:13.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:13.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:13.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:13.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:13.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:13.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:13.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:13.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:13.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:13.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:14.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:14.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:14.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:14.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:14.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:14.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:14.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:14.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:15.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:15.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:15.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:15.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:15.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:15.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:15.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:15.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:15.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:15.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:16.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:16.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:16.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:16.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:16.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:16.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:16.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:16.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:16.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:16.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:17.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:17.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:17.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:17.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:17.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:17.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:17.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:17.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:17.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:17.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:17.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:18.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:18.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:18.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:18.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:18.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:18.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:18.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:18.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:18.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:18.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:19.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:19.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:19.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:19.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:19.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:19.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:19.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:19.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:19.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:19.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:19.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:20.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:20.155 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-19 zones={test-zone2, test-zone1} 2021-03-12T02:35:20.156 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-19 zones={test-zone3, test-zone1} 2021-03-12T02:35:20.344 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:35:20.352 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:35:20.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.370 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:35:20.382 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:35:20.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.384 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.386 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.388 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.389 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.395 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:20.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:35:20.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:35:20.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:35:20.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:20.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:20.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:20.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:20.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:21.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:21.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:21.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:21.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:21.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:21.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:21.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:21.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:21.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:21.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:21.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:22.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:22.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:22.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:22.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:22.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:22.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:22.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:22.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:22.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:22.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:23.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:23.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:23.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:23.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:23.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:23.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:23.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:23.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:23.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:23.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:24.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:24.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:24.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:24.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:24.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:24.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:24.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:24.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:24.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:24.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:24.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:25.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:25.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:25.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:25.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:25.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:25.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:25.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:25.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:25.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:25.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:26.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:26.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:26.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:26.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:26.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:26.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:26.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:26.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:26.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:26.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:26.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:27.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:27.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:27.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:27.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:27.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:27.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:27.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:27.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:27.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:27.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:28.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:28.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:28.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:28.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:28.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:28.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:28.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:28.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:28.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:28.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:29.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:29.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:29.247 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-19 zones={test-zone3, test-zone1} 2021-03-12T02:35:29.247 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-19 zones={test-zone3, test-zone2} 2021-03-12T02:35:29.891 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:35:29.906 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:35:29.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.916 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.917 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.918 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.919 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:35:29.929 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-12T02:35:29.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.933 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.934 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.939 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.940 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.940 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.940 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.941 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.942 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.942 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:29.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:35:29.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:35:30.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-12T02:35:30.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:30.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:30.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:30.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:30.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:30.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:30.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:30.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:30.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:30.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:30.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:31.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:31.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:31.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:31.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:31.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:31.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:31.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:31.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:31.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:31.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:32.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:32.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:32.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:32.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:32.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:32.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:32.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:32.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:32.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:32.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:33.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:33.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:33.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:33.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:33.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:33.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:33.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:33.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:33.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:33.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:33.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:34.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:34.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:34.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:34.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:34.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:34.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:34.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:34.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:35.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:35.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:35.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:35.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:35.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:35.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:35.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:35.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:35.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:35.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:36.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:36.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:36.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:36.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:36.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:36.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:36.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:36.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:36.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:36.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:36.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:37.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:37.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:37.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:37.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:37.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:37.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:37.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:37.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:37.773 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:37.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:38.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:38.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:38.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:38.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:38.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:38.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:38.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:38.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:38.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:38.897 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-19 zones={test-zone3, test-zone2} 2021-03-12T02:35:38.898 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:38.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:38.899 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:38.939 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:38.940 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:39.140 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516456.957047_291.3","last_update":"2021-03-12T02:34:16.957047Z"},{"marker":"1_1615516500.963444_233.1","last_update":"2021-03-12T02:35:00.963444Z"},{"marker":"1_1615516500.963444_203.1","last_update":"2021-03-12T02:35:00.963444Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:35:39.140 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-12T02:35:39.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:39.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:39.142 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:39.186 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:39.186 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:39.377 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8376578737807985417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516500.963444_233.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:00.963444Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615516500.963444_203.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:00.963444Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:35:39.377 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-12T02:35:39.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:39.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:39.379 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:39.419 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:39.419 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:39.631 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516456.957047_291.3","last_update":"2021-03-12T02:34:16.957047Z"},{"marker":"1_1615516500.963444_233.1","last_update":"2021-03-12T02:35:00.963444Z"},{"marker":"1_1615516500.963444_203.1","last_update":"2021-03-12T02:35:00.963444Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:35:39.632 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-12T02:35:39.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:39.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:39.633 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:39.715 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:39.716 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:39.918 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1279455150295362088},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516500.963444_233.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:00.963444Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615516500.963444_203.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:00.963444Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:35:39.918 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-12T02:35:39.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:39.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:39.921 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:40.011 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:40.013 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:40.226 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516456.957047_291.3","last_update":"2021-03-12T02:34:16.957047Z"},{"marker":"1_1615516500.963444_233.1","last_update":"2021-03-12T02:35:00.963444Z"},{"marker":"1_1615516500.963444_203.1","last_update":"2021-03-12T02:35:00.963444Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:35:40.226 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-12T02:35:40.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:40.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:40.229 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:40.273 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:40.274 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:40.539 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11911835907256974194},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516500.963444_233.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:00.963444Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615516500.963444_203.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:00.963444Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:35:40.540 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-12T02:35:40.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:40.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:40.542 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:40.623 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:40.623 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:40.815 INFO:teuthology.orchestra.run.smithi114.stdout:[{"marker":"1_1615516457.862054_318.3","last_update":"2021-03-12T02:34:17.862054Z"},{"marker":"1_1615516501.868964_194.1","last_update":"2021-03-12T02:35:01.868964Z"},{"marker":"1_1615516501.868964_175.1","last_update":"2021-03-12T02:35:01.868964Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:35:40.816 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-12T02:35:40.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:40.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:40.817 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:40.862 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:40.862 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:41.072 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9936002905845503227},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516457.862054_318.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:17.862054Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516501.868964_194.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:01.868964Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615516501.868964_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:01.868964Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:35:41.072 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-12T02:35:41.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:41.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:41.074 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:41.113 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:41.114 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:41.305 INFO:teuthology.orchestra.run.smithi114.stdout:[{"marker":"1_1615516457.862054_318.3","last_update":"2021-03-12T02:34:17.862054Z"},{"marker":"1_1615516501.868964_194.1","last_update":"2021-03-12T02:35:01.868964Z"},{"marker":"1_1615516501.868964_175.1","last_update":"2021-03-12T02:35:01.868964Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:35:41.306 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-12T02:35:41.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:41.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:41.307 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:41.352 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:41.353 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:41.560 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3923957601198887203},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516457.862054_318.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:17.862054Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516501.868964_194.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:01.868964Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615516501.868964_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:01.868964Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:35:41.560 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-12T02:35:41.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:41.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:41.562 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:41.602 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:41.603 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:41.810 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516456.985100_243.3","last_update":"2021-03-12T02:34:16.985100Z"},{"marker":"1_1615516500.991879_153.1","last_update":"2021-03-12T02:35:00.991879Z"},{"marker":"1_1615516500.991879_145.1","last_update":"2021-03-12T02:35:00.991879Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:35:41.811 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-12T02:35:41.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:41.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:41.812 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:41.895 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:41.896 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:42.098 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12616054323329998860},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.985100_243.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.985100Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516500.991879_153.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:00.991879Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615516500.991879_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:00.991879Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:35:42.098 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-12T02:35:42.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:42.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:42.100 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:42.176 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:42.176 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:42.385 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516456.985100_243.3","last_update":"2021-03-12T02:34:16.985100Z"},{"marker":"1_1615516500.991879_153.1","last_update":"2021-03-12T02:35:00.991879Z"},{"marker":"1_1615516500.991879_145.1","last_update":"2021-03-12T02:35:00.991879Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:35:42.385 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-12T02:35:42.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:42.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:42.387 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:42.430 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:42.430 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:42.618 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10754379456686351807},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.985100_243.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.985100Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516500.991879_153.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:00.991879Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615516500.991879_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:00.991879Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:35:42.618 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-12T02:35:42.619 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2021-03-12T02:35:42.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:42.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:42.977 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:43.012 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:43.012 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:43.196 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:35:43.226 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516542.972050_191.1","last_update":"2021-03-12T02:35:42.972050Z"},{"marker":"1_1615516309.597935_168.1","last_update":"2021-03-12T02:31:49.597935Z"}] 2021-03-12T02:35:43.226 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:35:43.227 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:43.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:43.228 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:43.270 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:43.271 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:43.458 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516477.646256_184.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:37.646256Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:35:43.458 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615516542.972050_191.1 target=1_1615516477.646256_184.1 2021-03-12T02:35:48.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:48.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:48.461 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:48.502 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:48.502 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:48.693 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516542.972050_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:42.972050Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:35:48.694 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:35:48.694 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:35:48.695 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:48.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:48.701 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:48.744 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:48.745 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:48.952 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516542.972050_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:42.972050Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:35:48.953 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:35:48.953 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:35:48.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:48.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:48.955 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:48.996 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:48.997 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:49.255 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516542.972050_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:42.972050Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:35:49.256 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:35:49.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:49.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:49.593 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:49.635 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:49.635 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:49.831 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.28.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:35:49.831 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:35:49.832 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:49.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:49.833 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:49.875 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:49.875 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:50.077 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:35:50.078 WARNING:rgw_multi.tests:bucket bbucnh-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000001.28.5 target= 2021-03-12T02:35:55.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:55.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:55.082 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:55.126 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:55.127 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:55.334 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:35:55.335 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:35:55.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:55.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:55.338 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:55.382 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:55.382 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:55.579 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.28.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:35:55.579 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:35:55.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:55.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:55.581 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:55.667 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:55.667 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:55.899 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:35:55.900 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:35:55.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:55.901 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:55.901 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:55.982 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:55.983 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:56.201 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.28.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:35:56.201 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:35:56.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:56.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:56.203 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:56.247 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:56.508 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.28.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:35:56.509 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:35:56.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:56.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:56.511 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:56.587 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:56.587 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:56.772 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.42.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:35:56.772 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-20 2021-03-12T02:35:56.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:56.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:56.774 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:56.817 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:57.038 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:35:57.038 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-20 2021-03-12T02:35:57.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:57.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:57.040 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:57.085 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:57.086 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:57.261 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.42.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:35:57.262 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-20 2021-03-12T02:35:57.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:57.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:57.264 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:57.308 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:57.309 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:57.526 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:35:57.527 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-20 2021-03-12T02:35:57.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:57.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:57.529 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:57.570 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:57.571 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:57.773 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.23.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:35:57.773 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-20 2021-03-12T02:35:57.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:57.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:57.775 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:57.863 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:57.863 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:58.092 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.23.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:35:58.092 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-20 2021-03-12T02:35:58.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:58.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:58.094 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:58.177 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:58.178 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:58.396 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.23.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:35:58.397 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-20 2021-03-12T02:35:58.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:58.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:35:58.398 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:35:58.442 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:35:58.442 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:35:58.659 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.23.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:35:58.660 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-20 2021-03-12T02:35:58.660 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-20 zones={test-zone2, test-zone1} 2021-03-12T02:35:59.628 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:35:59.633 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:59.634 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:35:59.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:35:59.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:59.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:59.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:35:59.882 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-20 zones={test-zone2, test-zone1} 2021-03-12T02:35:59.883 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-20 zones={test-zone3, test-zone1} 2021-03-12T02:36:00.201 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:36:00.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:00.256 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:36:00.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:00.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:00.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:00.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:00.475 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-20 zones={test-zone3, test-zone1} 2021-03-12T02:36:00.475 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-20 zones={test-zone3, test-zone2} 2021-03-12T02:36:00.485 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:36:00.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:00.531 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:36:00.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:00.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:00.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:00.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:00.740 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-20 zones={test-zone3, test-zone2} 2021-03-12T02:36:00.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:00.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:00.755 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:00.796 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:00.796 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:00.970 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:36:01.002 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516560.750253_193.1","last_update":"2021-03-12T02:36:00.750253Z"},{"marker":"1_1615516309.597935_168.1","last_update":"2021-03-12T02:31:49.597935Z"}] 2021-03-12T02:36:01.003 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:36:01.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:01.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:01.004 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:01.048 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:01.048 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:01.234 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516560.741881_192.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:36:00.741881Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:36:01.235 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615516560.750253_193.1 target=1_1615516560.741881_192.1 2021-03-12T02:36:06.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:06.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:06.988 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:07.027 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:07.028 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:07.236 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516560.750253_193.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:36:00.750253Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:36:07.236 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:36:07.237 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:36:07.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:07.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:07.239 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:07.280 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:07.281 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:07.505 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516560.750253_193.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:36:00.750253Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:36:07.506 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:36:07.506 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:36:07.507 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:07.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:07.508 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:07.547 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:07.547 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:07.810 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516560.750253_193.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:36:00.750253Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:36:07.811 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:36:08.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:08.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:08.316 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:08.358 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:08.358 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:08.558 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.31.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:08.558 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:36:08.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:08.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:08.560 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:08.607 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:08.607 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:08.816 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:08.816 WARNING:rgw_multi.tests:bucket bbucnh-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.31.12 target=00000000001.28.5 2021-03-12T02:36:13.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:13.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:13.819 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:13.865 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:13.866 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:14.067 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.31.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:14.068 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:36:14.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:14.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:14.070 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:14.112 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:14.112 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:14.314 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.31.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:14.315 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:36:14.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:14.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:14.318 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:14.400 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:14.401 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:14.616 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.31.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:14.616 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:36:14.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:14.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:14.618 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:14.660 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:14.660 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:14.858 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.31.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:14.859 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:36:14.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:14.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:14.860 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:14.903 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:14.903 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:15.173 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.31.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:15.173 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:36:15.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:15.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:15.175 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:15.256 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:15.257 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:15.456 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.48.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:15.456 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-20 2021-03-12T02:36:15.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:15.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:15.458 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:15.503 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:15.503 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:15.727 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.48.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:15.727 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-20 2021-03-12T02:36:15.728 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:15.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:15.729 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:15.770 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:15.770 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:15.956 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.48.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:15.956 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-20 2021-03-12T02:36:15.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:15.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:15.958 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:16.002 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:16.002 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:16.221 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.48.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:16.222 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-20 2021-03-12T02:36:16.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:16.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:16.224 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:16.302 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:16.302 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:16.491 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.29.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:16.492 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-20 2021-03-12T02:36:16.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:16.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:16.494 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:16.583 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:16.584 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:16.807 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.29.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:16.807 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-20 2021-03-12T02:36:16.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:16.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:16.809 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:16.889 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:16.890 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:17.083 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.29.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:17.083 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-20 2021-03-12T02:36:17.084 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:17.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:17.085 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:17.129 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:17.130 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:17.333 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.29.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:17.334 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-20 2021-03-12T02:36:17.335 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-20 zones={test-zone2, test-zone1} 2021-03-12T02:36:17.856 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:36:17.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:17.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:17.863 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:36:17.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:17.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:17.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:17.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:18.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:18.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:18.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:18.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:18.242 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-20 zones={test-zone2, test-zone1} 2021-03-12T02:36:18.243 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-20 zones={test-zone3, test-zone1} 2021-03-12T02:36:18.793 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:36:18.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:18.847 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:18.848 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:36:18.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:18.855 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:18.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:18.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:18.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:19.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:19.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:19.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:19.197 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-20 zones={test-zone3, test-zone1} 2021-03-12T02:36:19.198 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-20 zones={test-zone3, test-zone2} 2021-03-12T02:36:19.956 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:36:19.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:19.965 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:19.965 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:36:20.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:20.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:20.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:20.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:20.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:20.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:20.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:20.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:20.384 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-20 zones={test-zone3, test-zone2} 2021-03-12T02:36:20.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:20.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:20.524 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:20.562 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:20.563 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:20.731 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:36:20.765 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516580.518628_195.1","last_update":"2021-03-12T02:36:20.518628Z"},{"marker":"1_1615516309.597935_168.1","last_update":"2021-03-12T02:31:49.597935Z"}] 2021-03-12T02:36:20.765 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:36:20.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:20.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:20.767 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:20.808 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:20.809 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:20.997 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516560.750253_193.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:36:00.750253Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:36:20.997 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615516580.518628_195.1 target=1_1615516560.750253_193.1 2021-03-12T02:36:25.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:25.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:26.000 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:26.040 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:26.041 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:26.241 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516580.518628_195.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:36:20.518628Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:36:26.242 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:36:26.242 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:36:26.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:26.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:26.244 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:26.290 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:26.291 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:26.465 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516580.518628_195.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:36:20.518628Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:36:26.466 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:36:26.467 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:36:26.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:26.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:26.468 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:26.508 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:26.508 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:26.763 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516580.518628_195.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:36:20.518628Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:36:26.764 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:36:27.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:27.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:27.304 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:27.342 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:27.342 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:27.543 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.35.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:27.544 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:36:27.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:27.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:27.545 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:27.585 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:27.788 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.31.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:27.788 WARNING:rgw_multi.tests:bucket bbucnh-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000005.35.17 target=00000000003.31.12 2021-03-12T02:36:32.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:35.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:35.832 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:35.879 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:35.879 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:36.090 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.35.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:36.091 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:36:36.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:36.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:36.094 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:36.136 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:36.136 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:36.323 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.35.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:36.324 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:36:36.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:36.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:36.325 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:36.409 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:36.409 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:36.628 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.35.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:36.629 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:36:36.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:36.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:36.630 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:36.714 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:36.714 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:36.919 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.35.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:36.919 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:36:36.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:36.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:36.921 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:36.966 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:36.967 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:37.227 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.35.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:37.227 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-20 2021-03-12T02:36:37.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:37.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:37.229 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:37.270 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:37.271 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:37.457 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.55.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:37.458 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-20 2021-03-12T02:36:37.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:37.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:37.459 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:37.503 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:37.504 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:37.710 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.55.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:37.710 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-20 2021-03-12T02:36:37.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:37.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:37.712 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:37.756 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:37.756 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:37.953 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.55.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:37.953 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-20 2021-03-12T02:36:37.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:37.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:37.955 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:38.001 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:38.213 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.55.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:38.214 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-20 2021-03-12T02:36:38.214 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:38.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:38.215 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:38.255 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:38.256 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:38.438 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.36.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:38.439 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-20 2021-03-12T02:36:38.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:38.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:38.440 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:38.524 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:38.524 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:38.744 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.36.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:38.745 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-20 2021-03-12T02:36:38.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:38.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:38.749 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:38.792 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:38.792 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:38.998 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.36.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:38.999 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-20 2021-03-12T02:36:39.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:39.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:39.001 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:39.048 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:39.048 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:39.243 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.36.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:39.244 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-20 2021-03-12T02:36:39.244 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-20 zones={test-zone2, test-zone1} 2021-03-12T02:36:39.642 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:36:39.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:39.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:39.653 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:36:39.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:39.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:39.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:39.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:39.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:39.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:40.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:40.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:40.148 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-20 zones={test-zone2, test-zone1} 2021-03-12T02:36:40.149 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-20 zones={test-zone3, test-zone1} 2021-03-12T02:36:40.236 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:36:40.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:40.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:40.288 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:36:40.297 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:40.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:40.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:40.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:40.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:40.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:40.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:40.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:40.838 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-20 zones={test-zone3, test-zone1} 2021-03-12T02:36:40.839 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-20 zones={test-zone3, test-zone2} 2021-03-12T02:36:40.851 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:36:40.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:40.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:40.904 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:36:40.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:40.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:40.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:40.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:41.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:41.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:41.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:41.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:41.268 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-20 zones={test-zone3, test-zone2} 2021-03-12T02:36:41.269 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2021-03-12T02:36:41.272 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-21 2021-03-12T02:36:41.308 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-22 2021-03-12T02:36:41.360 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-23 2021-03-12T02:36:41.443 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:41.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:41.446 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:41.486 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:41.487 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:41.665 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:36:41.696 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516601.435061_213.1","last_update":"2021-03-12T02:36:41.435061Z"},{"marker":"1_1615516309.597935_168.1","last_update":"2021-03-12T02:31:49.597935Z"}] 2021-03-12T02:36:41.697 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:36:41.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:41.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:41.699 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:41.755 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:41.756 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:41.921 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516601.435061_213.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:36:41.435061Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:36:41.922 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:36:41.923 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:36:41.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:41.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:41.926 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:41.966 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:41.967 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:42.169 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516601.435061_213.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:36:41.435061Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:36:42.169 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:36:42.170 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:36:42.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:42.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:42.171 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:42.211 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:42.211 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:42.474 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516601.435061_213.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:36:41.435061Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:36:42.474 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:36:44.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:44.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:44.428 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:44.465 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:44.466 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:44.677 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.17.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:44.678 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-21 2021-03-12T02:36:44.678 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:44.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:44.679 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:44.724 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:44.724 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:44.913 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"mpLFS9J8P4f-7b6R3VBPCpivOrJjC.7","ns":""},"count":1},"inc_marker":{"position":"00000000003.17.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:44.913 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-21 2021-03-12T02:36:44.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:44.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:44.915 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:44.958 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:44.958 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:45.149 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.17.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:45.149 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-21 2021-03-12T02:36:45.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:45.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:45.151 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:45.232 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:45.232 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:45.448 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"mpLFS9J8P4f-7b6R3VBPCpivOrJjC.7","ns":""},"count":1},"inc_marker":{"position":"00000000003.17.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:45.448 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-21 2021-03-12T02:36:45.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:45.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:45.450 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:45.531 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:45.531 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:45.730 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.17.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:45.730 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-21 2021-03-12T02:36:45.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:45.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:45.732 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:45.775 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:45.775 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:46.037 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.17.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:46.037 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-21 2021-03-12T02:36:46.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:46.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:46.039 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:46.120 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:46.121 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:46.303 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.22.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:46.304 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-21 2021-03-12T02:36:46.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:46.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:46.305 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:46.349 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:46.349 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:46.561 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"mpLFS9J8P4f-7b6R3VBPCpivOrJjC.7","ns":""},"count":1},"inc_marker":{"position":"00000000002.22.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:46.562 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-21 2021-03-12T02:36:46.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:46.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:46.565 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:46.609 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:46.609 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:46.791 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.22.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:46.791 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-21 2021-03-12T02:36:46.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:46.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:46.794 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:46.837 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:46.837 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:47.039 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"mpLFS9J8P4f-7b6R3VBPCpivOrJjC.7","ns":""},"count":1},"inc_marker":{"position":"00000000002.22.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:47.040 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-21 2021-03-12T02:36:47.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:47.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:47.042 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:47.123 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:47.124 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:47.321 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.24.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:47.321 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-21 2021-03-12T02:36:47.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:47.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:47.323 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:47.405 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:47.405 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:47.635 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"mpLFS9J8P4f-7b6R3VBPCpivOrJjC.7","ns":""},"count":1},"inc_marker":{"position":"00000000002.24.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:47.635 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-21 2021-03-12T02:36:47.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:47.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:47.637 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:47.721 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:47.722 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:47.926 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.24.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:47.926 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-21 2021-03-12T02:36:47.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:47.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:47.928 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:47.974 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:47.975 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:48.186 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"mpLFS9J8P4f-7b6R3VBPCpivOrJjC.7","ns":""},"count":1},"inc_marker":{"position":"00000000002.24.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:48.186 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-21 2021-03-12T02:36:48.187 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-21 zones={test-zone2, test-zone1} 2021-03-12T02:36:49.461 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:36:49.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:49.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:49.469 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:36:49.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:49.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:49.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:49.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:49.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:49.666 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-21 zones={test-zone2, test-zone1} 2021-03-12T02:36:49.666 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-21 zones={test-zone3, test-zone1} 2021-03-12T02:36:50.066 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:36:50.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:50.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:50.116 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:36:50.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:50.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:50.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:50.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:50.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:50.269 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-21 zones={test-zone3, test-zone1} 2021-03-12T02:36:50.270 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-21 zones={test-zone3, test-zone2} 2021-03-12T02:36:50.279 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:36:50.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:50.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:50.332 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:36:50.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:50.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:50.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:50.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:50.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:50.484 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-21 zones={test-zone3, test-zone2} 2021-03-12T02:36:50.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:50.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:50.486 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:50.528 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:50.528 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:50.740 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.21.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:50.741 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-22 2021-03-12T02:36:50.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:50.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:50.743 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:50.786 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:50.787 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:51.000 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"dFNlvqVW69ugNd1KkluGu1aglGvaV4D","ns":""},"count":1},"inc_marker":{"position":"00000000002.21.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:51.001 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-22 2021-03-12T02:36:51.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:51.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:51.003 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:51.044 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:51.268 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.21.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:51.269 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-22 2021-03-12T02:36:51.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:51.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:51.270 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:51.349 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:51.349 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:51.562 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"dFNlvqVW69ugNd1KkluGu1aglGvaV4D","ns":""},"count":1},"inc_marker":{"position":"00000000002.21.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:51.563 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-22 2021-03-12T02:36:51.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:51.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:51.565 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:51.642 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:51.642 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:51.846 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.21.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:51.846 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-22 2021-03-12T02:36:51.847 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:51.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:51.848 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:51.892 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:51.893 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:52.150 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:52.151 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-22 2021-03-12T02:36:52.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:52.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:52.152 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:52.234 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:52.434 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.27.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:52.435 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-22 2021-03-12T02:36:52.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:52.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:52.436 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:52.480 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:52.480 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:52.727 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"dFNlvqVW69ugNd1KkluGu1aglGvaV4D","ns":""},"count":1},"inc_marker":{"position":"00000000003.27.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:52.728 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-22 2021-03-12T02:36:52.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:52.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:52.731 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:52.774 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:52.775 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:52.978 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.27.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:52.979 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-22 2021-03-12T02:36:52.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:52.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:52.981 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:53.025 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:53.025 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:53.246 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"dFNlvqVW69ugNd1KkluGu1aglGvaV4D","ns":""},"count":1},"inc_marker":{"position":"00000000003.27.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:53.247 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-22 2021-03-12T02:36:53.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:53.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:53.249 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:53.328 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:53.329 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:53.548 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.22.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:53.549 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-22 2021-03-12T02:36:53.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:53.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:53.550 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:53.868 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"dFNlvqVW69ugNd1KkluGu1aglGvaV4D","ns":""},"count":1},"inc_marker":{"position":"00000000002.22.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:53.868 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-22 2021-03-12T02:36:53.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:53.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:53.870 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:53.951 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:53.951 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:54.155 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.22.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:54.155 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-22 2021-03-12T02:36:54.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:54.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:54.157 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:54.201 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:54.202 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:54.429 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"dFNlvqVW69ugNd1KkluGu1aglGvaV4D","ns":""},"count":1},"inc_marker":{"position":"00000000002.22.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:54.429 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-22 2021-03-12T02:36:54.430 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-22 zones={test-zone2, test-zone1} 2021-03-12T02:36:55.898 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:36:55.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:55.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:55.906 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:36:55.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:55.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:56.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:56.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:56.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:56.096 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-22 zones={test-zone2, test-zone1} 2021-03-12T02:36:56.096 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-22 zones={test-zone3, test-zone1} 2021-03-12T02:36:56.706 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:36:56.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:56.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:56.756 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:36:56.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:56.763 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:56.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:56.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:56.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:56.903 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-22 zones={test-zone3, test-zone1} 2021-03-12T02:36:56.904 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-22 zones={test-zone3, test-zone2} 2021-03-12T02:36:56.912 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:36:56.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:56.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:56.964 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:36:56.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:56.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:36:57.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:57.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:57.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:36:57.123 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-22 zones={test-zone3, test-zone2} 2021-03-12T02:36:57.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:57.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:57.124 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:57.166 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:57.166 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:57.365 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.44.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:57.366 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-23 2021-03-12T02:36:57.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:57.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:57.367 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:57.410 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:57.411 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:57.600 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"sUMbJTaOIm-o.5JWXGS4-0IiDKx9Zh-","ns":""},"count":2},"inc_marker":{"position":"00000000002.44.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:57.601 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-23 2021-03-12T02:36:57.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:57.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:57.602 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:57.642 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:57.643 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:57.853 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.44.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:57.854 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-23 2021-03-12T02:36:57.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:57.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:57.855 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:57.937 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:58.170 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"sUMbJTaOIm-o.5JWXGS4-0IiDKx9Zh-","ns":""},"count":2},"inc_marker":{"position":"00000000002.44.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:58.170 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-23 2021-03-12T02:36:58.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:58.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:58.172 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:58.250 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:58.250 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:58.459 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.44.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:58.460 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-23 2021-03-12T02:36:58.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:58.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:58.461 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:58.506 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:58.506 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:58.761 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.44.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:58.762 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-23 2021-03-12T02:36:58.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:58.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:58.763 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:58.805 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:58.805 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:59.000 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.28.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:59.001 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-23 2021-03-12T02:36:59.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:59.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:59.002 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:59.046 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:59.047 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:59.285 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"sUMbJTaOIm-o.5JWXGS4-0IiDKx9Zh-","ns":""},"count":2},"inc_marker":{"position":"00000000002.28.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:59.286 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-23 2021-03-12T02:36:59.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:59.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:59.288 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:59.328 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:59.328 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:59.519 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.28.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:36:59.519 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-23 2021-03-12T02:36:59.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:59.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:59.521 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:59.565 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:59.565 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:36:59.781 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"sUMbJTaOIm-o.5JWXGS4-0IiDKx9Zh-","ns":""},"count":2},"inc_marker":{"position":"00000000002.28.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:36:59.782 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-23 2021-03-12T02:36:59.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:59.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:36:59.784 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:36:59.866 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:36:59.866 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:00.075 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.28.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:00.076 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-23 2021-03-12T02:37:00.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:00.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:00.077 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:00.161 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:00.161 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:00.393 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"sUMbJTaOIm-o.5JWXGS4-0IiDKx9Zh-","ns":""},"count":2},"inc_marker":{"position":"00000000003.28.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:00.393 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-23 2021-03-12T02:37:00.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:00.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:00.395 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:00.474 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:00.475 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:00.691 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.28.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:00.691 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-23 2021-03-12T02:37:00.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:00.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:00.693 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:00.737 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:00.737 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:00.939 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"sUMbJTaOIm-o.5JWXGS4-0IiDKx9Zh-","ns":""},"count":2},"inc_marker":{"position":"00000000003.28.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:00.940 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-23 2021-03-12T02:37:00.940 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-23 zones={test-zone2, test-zone1} 2021-03-12T02:37:02.841 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:37:02.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:02.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:02.850 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:37:02.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:02.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:02.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:02.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:02.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:03.044 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-23 zones={test-zone2, test-zone1} 2021-03-12T02:37:03.045 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-23 zones={test-zone3, test-zone1} 2021-03-12T02:37:03.470 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:37:03.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:03.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:03.524 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:37:03.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:03.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:03.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:03.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:03.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:03.674 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-23 zones={test-zone3, test-zone1} 2021-03-12T02:37:03.674 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-23 zones={test-zone3, test-zone2} 2021-03-12T02:37:03.683 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:37:03.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:03.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:03.736 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:37:03.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:03.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:03.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:03.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:03.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:03.887 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-23 zones={test-zone3, test-zone2} 2021-03-12T02:37:03.888 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2021-03-12T02:37:03.889 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-24 2021-03-12T02:37:03.921 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-25 2021-03-12T02:37:03.977 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-26 2021-03-12T02:37:04.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:04.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:04.133 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:04.172 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:04.172 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:04.336 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:37:04.369 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516624.119886_249.1","last_update":"2021-03-12T02:37:04.119886Z"},{"marker":"1_1615516309.597935_168.1","last_update":"2021-03-12T02:31:49.597935Z"}] 2021-03-12T02:37:04.369 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:37:04.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:04.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:04.371 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:04.411 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:04.412 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:04.607 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516624.080300_245.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:04.080300Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:37:04.607 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615516624.119886_249.1 target=1_1615516624.080300_245.1 2021-03-12T02:37:09.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:09.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:09.610 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:09.650 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:09.650 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:09.853 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516624.119886_249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:04.119886Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:37:09.854 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:37:09.854 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:37:09.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:09.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:09.856 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:09.897 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:09.897 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:10.112 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516624.119886_249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:04.119886Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:37:10.113 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:37:10.113 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:37:10.114 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:10.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:10.115 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:10.155 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:10.155 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:10.418 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516624.119886_249.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:04.119886Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:37:10.611 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:37:12.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:12.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:12.208 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:12.247 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:12.248 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:12.440 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.42.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:12.441 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-24 2021-03-12T02:37:12.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:12.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:12.444 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:12.491 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:12.492 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:12.693 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.42.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:12.694 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-24 2021-03-12T02:37:12.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:12.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:12.696 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:12.736 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:12.737 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:12.953 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.42.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:12.953 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-24 2021-03-12T02:37:12.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:12.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:12.955 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:13.037 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:13.038 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:13.276 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.42.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:13.277 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-24 2021-03-12T02:37:13.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:13.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:13.278 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:13.356 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:13.356 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:13.551 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.42.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:13.551 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-24 2021-03-12T02:37:13.552 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:13.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:13.553 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:13.597 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:13.597 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:13.861 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.42.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:13.862 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-24 2021-03-12T02:37:13.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:13.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:13.864 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:13.905 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:13.905 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:14.100 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.34.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:14.100 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-24 2021-03-12T02:37:14.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:14.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:14.102 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:14.140 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:14.140 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:14.330 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.34.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:14.331 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-24 2021-03-12T02:37:14.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:14.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:14.332 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:14.375 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:14.375 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:14.569 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.34.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:14.569 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-24 2021-03-12T02:37:14.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:14.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:14.571 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:14.616 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:14.616 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:14.845 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.34.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:14.845 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-24 2021-03-12T02:37:14.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:14.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:14.847 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:14.890 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:14.890 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:15.095 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.53.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:15.096 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-24 2021-03-12T02:37:15.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:15.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:15.097 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:15.181 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:15.182 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:15.391 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:15.392 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-24 2021-03-12T02:37:15.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:15.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:15.397 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:15.477 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:15.478 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:15.693 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.53.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:15.694 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-24 2021-03-12T02:37:15.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:15.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:15.696 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:15.743 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:15.743 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:15.948 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:15.949 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-24 2021-03-12T02:37:15.950 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-24 zones={test-zone2, test-zone1} 2021-03-12T02:37:16.559 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:37:16.565 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:16.566 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:37:16.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:16.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:16.680 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-24 zones={test-zone2, test-zone1} 2021-03-12T02:37:16.681 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-24 zones={test-zone3, test-zone1} 2021-03-12T02:37:16.820 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:37:16.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:16.867 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:37:16.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:16.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:16.935 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-24 zones={test-zone3, test-zone1} 2021-03-12T02:37:16.935 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-24 zones={test-zone3, test-zone2} 2021-03-12T02:37:16.944 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:37:16.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:16.991 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:37:16.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:17.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:17.059 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-24 zones={test-zone3, test-zone2} 2021-03-12T02:37:17.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:17.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:17.062 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:17.105 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:17.105 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:17.302 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.42.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:17.303 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-25 2021-03-12T02:37:17.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:17.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:17.306 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:17.352 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:17.352 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:17.544 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:17.545 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-25 2021-03-12T02:37:17.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:17.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:17.546 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:17.586 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:17.587 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:17.813 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.42.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:17.813 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-25 2021-03-12T02:37:17.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:17.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:17.815 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:17.897 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:17.897 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:18.125 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:18.126 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-25 2021-03-12T02:37:18.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:18.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:18.128 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:18.208 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:18.209 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:18.417 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.42.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:18.418 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-25 2021-03-12T02:37:18.418 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:18.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:18.419 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:18.462 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:18.463 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:18.730 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.42.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:18.731 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-25 2021-03-12T02:37:18.731 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:18.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:18.732 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:18.812 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:18.812 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:19.001 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.76.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:19.002 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-25 2021-03-12T02:37:19.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:19.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:19.004 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:19.049 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:19.050 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:19.264 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.76.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:19.265 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-25 2021-03-12T02:37:19.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:19.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:19.267 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:19.307 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:19.308 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:19.499 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.76.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:19.500 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-25 2021-03-12T02:37:19.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:19.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:19.503 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:19.549 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:19.549 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:19.768 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.76.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:19.769 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-25 2021-03-12T02:37:19.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:19.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:19.771 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:19.851 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:19.851 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:20.064 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.50.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:20.065 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-25 2021-03-12T02:37:20.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:20.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:20.066 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:20.151 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:20.152 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:20.383 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:20.383 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-25 2021-03-12T02:37:20.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:20.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:20.385 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:20.464 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:20.465 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:20.683 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.50.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:20.684 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-25 2021-03-12T02:37:20.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:20.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:20.685 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:20.730 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:20.730 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:20.935 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:20.935 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-25 2021-03-12T02:37:20.936 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-25 zones={test-zone2, test-zone1} 2021-03-12T02:37:21.377 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:37:21.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:21.384 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:37:21.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:21.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:21.503 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-25 zones={test-zone2, test-zone1} 2021-03-12T02:37:21.504 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-25 zones={test-zone3, test-zone1} 2021-03-12T02:37:22.383 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:37:22.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:22.435 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:37:22.442 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:22.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:22.503 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-25 zones={test-zone3, test-zone1} 2021-03-12T02:37:22.504 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-25 zones={test-zone3, test-zone2} 2021-03-12T02:37:22.658 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:37:22.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:22.669 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:37:22.722 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:22.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:22.788 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-25 zones={test-zone3, test-zone2} 2021-03-12T02:37:22.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:22.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:22.790 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:22.828 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:22.829 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:23.030 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.12.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:23.030 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-26 2021-03-12T02:37:23.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:23.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:23.032 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:23.076 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:23.076 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:23.287 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:23.287 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-26 2021-03-12T02:37:23.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:23.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:23.289 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:23.332 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:23.332 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:23.544 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.12.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:23.544 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-26 2021-03-12T02:37:23.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:23.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:23.546 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:23.632 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:23.632 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:23.852 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:23.853 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-26 2021-03-12T02:37:23.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:23.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:23.855 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:23.896 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:23.897 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:24.099 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.12.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:24.100 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-26 2021-03-12T02:37:24.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:24.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:24.102 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:24.148 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:24.148 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:24.408 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.12.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:24.409 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-26 2021-03-12T02:37:24.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:24.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:24.411 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:24.491 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:24.492 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:24.688 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.19.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:24.689 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-26 2021-03-12T02:37:24.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:24.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:24.691 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:24.735 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:24.735 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:24.955 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.19.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:24.956 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-26 2021-03-12T02:37:24.957 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:24.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:24.958 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:25.000 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:25.001 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:25.193 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.19.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:25.193 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-26 2021-03-12T02:37:25.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:25.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:25.195 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:25.239 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:25.239 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:25.455 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.19.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:25.456 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-26 2021-03-12T02:37:25.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:25.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:25.457 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:25.538 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:25.538 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:25.749 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.24.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:25.749 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-26 2021-03-12T02:37:25.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:25.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:25.751 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:25.833 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:25.833 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:26.063 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.24.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:26.064 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-26 2021-03-12T02:37:26.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:26.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:26.065 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:26.146 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:26.147 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:26.357 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.24.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:37:26.358 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-26 2021-03-12T02:37:26.359 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:26.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:26.360 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:26.402 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:26.403 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:26.591 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.24.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:37:26.592 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-26 2021-03-12T02:37:26.592 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-26 zones={test-zone2, test-zone1} 2021-03-12T02:37:27.965 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:37:27.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:27.971 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:37:28.022 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:28.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:28.084 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-26 zones={test-zone2, test-zone1} 2021-03-12T02:37:28.085 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-26 zones={test-zone3, test-zone1} 2021-03-12T02:37:28.688 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:37:28.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:28.735 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:37:28.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:28.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:28.802 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-26 zones={test-zone3, test-zone1} 2021-03-12T02:37:28.802 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-26 zones={test-zone3, test-zone2} 2021-03-12T02:37:28.811 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:37:28.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:28.859 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:37:28.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-12T02:37:28.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-12T02:37:28.930 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-26 zones={test-zone3, test-zone2} 2021-03-12T02:37:28.935 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2021-03-12T02:37:28.936 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-27 2021-03-12T02:37:28.968 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-28 2021-03-12T02:37:29.023 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-29 2021-03-12T02:37:29.258 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2021-03-12T02:37:29.259 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-30 2021-03-12T02:37:29.290 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-31 2021-03-12T02:37:29.347 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-32 2021-03-12T02:37:29.591 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2021-03-12T02:37:29.592 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-33 2021-03-12T02:37:29.624 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-34 2021-03-12T02:37:29.681 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-35 2021-03-12T02:37:32.399 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2021-03-12T02:37:32.399 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-36 2021-03-12T02:37:33.086 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-37 2021-03-12T02:37:33.700 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-38 2021-03-12T02:37:34.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:34.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:34.032 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:34.069 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:34.069 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:34.241 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:37:34.270 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516654.008644_327.1","last_update":"2021-03-12T02:37:34.008644Z"},{"marker":"1_1615516309.597935_168.1","last_update":"2021-03-12T02:31:49.597935Z"}] 2021-03-12T02:37:34.271 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:37:34.271 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:34.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:34.272 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:34.311 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:34.311 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:34.512 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516654.008644_327.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:34.008644Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:37:34.513 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:37:34.514 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:37:34.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:34.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:34.516 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:34.558 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:34.558 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:34.787 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516654.008644_327.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:34.008644Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:37:34.788 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:37:34.788 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:37:34.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:34.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:34.790 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:34.829 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:34.829 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:35.189 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516654.008644_327.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:34.008644Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:37:35.189 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:37:36.460 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2021-03-12T02:37:36.460 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-39 2021-03-12T02:37:36.488 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-40 2021-03-12T02:37:36.549 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-41 2021-03-12T02:37:36.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:36.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:36.605 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:36.641 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:36.642 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:36.827 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=0cf3ee11-29e6-4776-941b-50bda62adf82 2021-03-12T02:37:36.853 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516156.395861_98.1","last_update":"2021-03-12T02:29:16.395861Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516656.580426_339.1","last_update":"2021-03-12T02:37:36.580426Z"},{"marker":"1_1615516309.597935_168.1","last_update":"2021-03-12T02:31:49.597935Z"}] 2021-03-12T02:37:36.854 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:37:36.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:36.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:36.855 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:36.892 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:36.893 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:37.081 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516654.008644_327.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:34.008644Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:37:37.081 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615516656.580426_339.1 target=1_1615516654.008644_327.1 2021-03-12T02:37:42.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:42.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:42.084 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:42.124 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:42.125 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:42.317 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516656.580426_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:36.580426Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:37:42.317 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:37:42.318 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:37:42.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:42.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:42.320 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:42.358 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:42.358 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:42.564 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516656.580426_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:36.580426Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:37:42.565 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:37:42.565 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:37:42.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:42.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:37:42.567 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:37:42.607 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:42.607 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:42.872 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516656.580426_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:36.580426Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:37:42.873 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:37:42.873 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-03-12T02:37:42.874 INFO:teuthology.orchestra.run:waiting for 300 2021-03-12T02:37:42.875 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:37:42.868+0000 7f1e8e77d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends civetweb port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 17924) UID: 0 2021-03-12T02:37:42.875 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:37:42.868+0000 7f1ea5e410c0 -1 shutting down 2021-03-12T02:37:44.500 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-03-12T02:37:48.975 INFO:tasks.rgw.c1.client.1:Stopped 2021-03-12T02:37:48.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:37:48.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:37:48.977 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:37:49.017 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:49.018 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:49.045 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:37:49.043+0000 7f3de49be080 0 NOTICE: overriding master zone: 6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e 2021-03-12T02:37:49.047 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:37:49.043+0000 7f3de49be080 0 failed reading obj info from .rgw.root:zone_info.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e: (2) No such file or directory 2021-03-12T02:37:49.047 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:37:49.043+0000 7f3de49be080 0 WARNING: could not read zone params for zone id=6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e name=test-zone1 2021-03-12T02:37:49.048 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:37:49.043+0000 7f3de49be080 0 failed reading obj info from .rgw.root:zone_info.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107: (2) No such file or directory 2021-03-12T02:37:49.049 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:37:49.043+0000 7f3de49be080 0 WARNING: could not read zone params for zone id=e98c2a8b-a28f-47cd-a0ca-2777b6b4f107 name=test-zone3 2021-03-12T02:37:49.060 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"IMFN8A2KI1WM97OP","secret_key":"SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"0428d020-efe8-4edc-a347-2def7c927539"} 2021-03-12T02:37:49.061 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:37:49.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:37:49.062 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:37:49.143 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:37:49.143 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:37:49.396 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"76eba794-fdc5-4e5e-9ce6-5f5a18a97795","epoch":1,"predecessor_uuid":"0cf3ee11-29e6-4776-941b-50bda62adf82","sync_status":["1_1615516156.395861_98.1","","1_1615516656.580426_339.1","1_1615516309.597935_168.1"],"period_map":{"id":"76eba794-fdc5-4e5e-9ce6-5f5a18a97795","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"374a14e6-b245-4495-acdf-8a1b3996ba1d","zones":[{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","endpoints":["http://smithi114.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","name":"test-zone4","endpoints":["http://smithi114.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","name":"test-zone3","endpoints":["http://smithi077.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"374a14e6-b245-4495-acdf-8a1b3996ba1d","val":3185083755},{"key":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","val":2537029094},{"key":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","val":3794530578},{"key":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","val":3019679815}]},"master_zonegroup":"test-zonegroup","master_zone":"374a14e6-b245-4495-acdf-8a1b3996ba1d","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"0428d020-efe8-4edc-a347-2def7c927539","realm_name":"test-realm","realm_epoch":3} 2021-03-12T02:37:49.397 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2021-03-12T02:37:50.106 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-03-12T02:38:49.399 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-42 2021-03-12T02:38:49.465 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-43 2021-03-12T02:38:49.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:38:49.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:38:49.495 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:38:49.530 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:38:49.530 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:38:49.685 INFO:teuthology.orchestra.run.smithi114.stderr:No --period given, using current period=76eba794-fdc5-4e5e-9ce6-5f5a18a97795 2021-03-12T02:38:49.715 INFO:teuthology.orchestra.run.smithi114.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516729.491953_32.1","last_update":"2021-03-12T02:38:49.491953Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:38:49.716 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2021-03-12T02:38:49.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:38:49.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:38:49.717 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:38:49.758 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:38:49.758 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:38:49.962 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"76eba794-fdc5-4e5e-9ce6-5f5a18a97795","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":80,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":80,"complete":80}} 2021-03-12T02:38:49.963 WARNING:rgw_multi.tests:zone test-zone1 behind master: shard=2 master=1_1615516729.491953_32.1 target= 2021-03-12T02:38:54.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:38:54.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:38:54.965 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:38:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:38:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:38:55.210 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"76eba794-fdc5-4e5e-9ce6-5f5a18a97795","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":80,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"1_1615516729.491953_32.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:38:49.491953Z","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":80,"complete":80}} 2021-03-12T02:38:55.211 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2021-03-12T02:38:55.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:38:55.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:38:55.213 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:38:55.292 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:38:55.293 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:38:55.320 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:38:55.310+0000 7f726fde6080 0 NOTICE: overriding master zone: 374a14e6-b245-4495-acdf-8a1b3996ba1d 2021-03-12T02:38:55.321 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:38:55.314+0000 7f726fde6080 0 failed reading obj info from .rgw.root:zone_info.374a14e6-b245-4495-acdf-8a1b3996ba1d: (2) No such file or directory 2021-03-12T02:38:55.321 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:38:55.314+0000 7f726fde6080 0 WARNING: could not read zone params for zone id=374a14e6-b245-4495-acdf-8a1b3996ba1d name=test-zone2 2021-03-12T02:38:55.322 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:38:55.314+0000 7f726fde6080 0 failed reading obj info from .rgw.root:zone_info.ad4e1236-7627-4342-9dd1-46e3c42d2e72: (2) No such file or directory 2021-03-12T02:38:55.323 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:38:55.314+0000 7f726fde6080 0 WARNING: could not read zone params for zone id=ad4e1236-7627-4342-9dd1-46e3c42d2e72 name=test-zone4 2021-03-12T02:38:55.333 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"IMFN8A2KI1WM97OP","secret_key":"SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"0428d020-efe8-4edc-a347-2def7c927539"} 2021-03-12T02:38:55.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:38:55.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:38:55.335 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:38:55.417 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:38:55.417 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:38:55.690 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","epoch":1,"predecessor_uuid":"76eba794-fdc5-4e5e-9ce6-5f5a18a97795","sync_status":["","","1_1615516729.491953_32.1",""],"period_map":{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","zones":[{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","endpoints":["http://smithi114.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","name":"test-zone4","endpoints":["http://smithi114.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","name":"test-zone3","endpoints":["http://smithi077.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"374a14e6-b245-4495-acdf-8a1b3996ba1d","val":3185083755},{"key":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","val":2537029094},{"key":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","val":3794530578},{"key":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","val":3019679815}]},"master_zonegroup":"test-zonegroup","master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"0428d020-efe8-4edc-a347-2def7c927539","realm_name":"test-realm","realm_epoch":4} 2021-03-12T02:38:55.691 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2021-03-12T02:39:55.692 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-44 2021-03-12T02:39:55.737 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-45 2021-03-12T02:39:55.804 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-03-12T02:39:55.805 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'civetweb port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-03-12T02:39:55.809 INFO:tasks.rgw.c1.client.1:Started 2021-03-12T02:39:55.809 DEBUG:teuthology.orchestra.run.smithi077:> curl http://smithi077.front.sepia.ceph.com:8001 2021-03-12T02:39:55.824 INFO:teuthology.orchestra.run.smithi077.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-12T02:39:55.825 INFO:teuthology.orchestra.run.smithi077.stderr: Dload Upload Total Spent Left Speed 2021-03-12T02:39:55.827 INFO:teuthology.orchestra.run.smithi077.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi077.front.sepia.ceph.com port 8001: Connection refused 2021-03-12T02:39:55.828 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-03-12T02:39:56.830 DEBUG:teuthology.orchestra.run.smithi077:> curl http://smithi077.front.sepia.ceph.com:8001 2021-03-12T02:39:56.847 INFO:teuthology.orchestra.run.smithi077.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-12T02:39:56.847 INFO:teuthology.orchestra.run.smithi077.stderr: Dload Upload Total Spent Left Speed 2021-03-12T02:39:56.853 INFO:teuthology.orchestra.run.smithi077.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 53500 0 --:--:-- --:--:-- --:--:-- 53500 2021-03-12T02:39:56.854 INFO:teuthology.orchestra.run.smithi077.stdout:anonymous 2021-03-12T02:39:56.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:39:56.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:39:56.859 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:39:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:39:56.939 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:39:57.127 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:39:57.161 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516795.780925_564.1","last_update":"2021-03-12T02:39:55.780925Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:39:57.162 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:39:57.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:39:57.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:39:57.163 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:39:57.203 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:39:57.204 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:39:57.421 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"0cf3ee11-29e6-4776-941b-50bda62adf82","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516656.580426_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:36.580426Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:39:57.422 WARNING:rgw_multi.tests:zone test-zone2 is syncing realm epoch=2, behind current realm epoch=4 2021-03-12T02:40:02.423 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:02.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:02.425 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:02.465 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:02.465 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:02.653 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516795.780925_564.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:39:55.780925Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:40:02.654 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:40:02.655 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:40:02.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:02.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:02.656 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:02.697 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:02.697 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:02.906 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516795.780925_564.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:39:55.780925Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:40:02.906 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:40:02.907 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:40:02.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:02.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:02.909 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:02.949 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:02.949 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:03.213 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516795.780925_564.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:39:55.780925Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:40:03.214 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:40:03.214 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-39 zones={test-zone2, test-zone1} 2021-03-12T02:40:04.678 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:04.685 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:04.794 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-39 zones={test-zone2, test-zone1} 2021-03-12T02:40:04.795 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-39 zones={test-zone3, test-zone1} 2021-03-12T02:40:04.814 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:04.863 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:04.926 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-39 zones={test-zone3, test-zone1} 2021-03-12T02:40:04.927 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-39 zones={test-zone1, test-zone2} 2021-03-12T02:40:04.936 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:04.983 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:05.048 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-39 zones={test-zone1, test-zone2} 2021-03-12T02:40:05.048 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-39 zones={test-zone3, test-zone2} 2021-03-12T02:40:05.058 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:05.106 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:05.168 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-39 zones={test-zone3, test-zone2} 2021-03-12T02:40:05.169 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-39 zones={test-zone1, test-zone3} 2021-03-12T02:40:05.181 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:05.235 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:05.300 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-39 zones={test-zone1, test-zone3} 2021-03-12T02:40:05.301 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-39 zones={test-zone2, test-zone3} 2021-03-12T02:40:05.311 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:05.363 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:05.431 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-39 zones={test-zone2, test-zone3} 2021-03-12T02:40:05.431 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-40 zones={test-zone2, test-zone1} 2021-03-12T02:40:05.443 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:05.491 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:05.555 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-40 zones={test-zone2, test-zone1} 2021-03-12T02:40:05.555 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-40 zones={test-zone3, test-zone1} 2021-03-12T02:40:05.564 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:05.614 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:05.682 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-40 zones={test-zone3, test-zone1} 2021-03-12T02:40:05.683 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-40 zones={test-zone1, test-zone2} 2021-03-12T02:40:05.695 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:05.751 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:05.822 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-40 zones={test-zone1, test-zone2} 2021-03-12T02:40:05.823 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-40 zones={test-zone3, test-zone2} 2021-03-12T02:40:05.833 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:05.883 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:05.944 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-40 zones={test-zone3, test-zone2} 2021-03-12T02:40:05.945 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-40 zones={test-zone1, test-zone3} 2021-03-12T02:40:05.957 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:06.003 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:06.065 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-40 zones={test-zone1, test-zone3} 2021-03-12T02:40:06.066 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-40 zones={test-zone2, test-zone3} 2021-03-12T02:40:06.078 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:06.127 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:06.194 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-40 zones={test-zone2, test-zone3} 2021-03-12T02:40:06.195 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-41 zones={test-zone2, test-zone1} 2021-03-12T02:40:06.207 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:06.259 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:06.326 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-41 zones={test-zone2, test-zone1} 2021-03-12T02:40:06.327 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-41 zones={test-zone3, test-zone1} 2021-03-12T02:40:06.338 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:06.387 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:06.450 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-41 zones={test-zone3, test-zone1} 2021-03-12T02:40:06.450 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-41 zones={test-zone1, test-zone2} 2021-03-12T02:40:06.460 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:06.511 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:06.573 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-41 zones={test-zone1, test-zone2} 2021-03-12T02:40:06.574 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-41 zones={test-zone3, test-zone2} 2021-03-12T02:40:06.590 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:06.638 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:06.705 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-41 zones={test-zone3, test-zone2} 2021-03-12T02:40:06.706 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-41 zones={test-zone1, test-zone3} 2021-03-12T02:40:06.718 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:06.767 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:06.831 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-41 zones={test-zone1, test-zone3} 2021-03-12T02:40:06.831 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-41 zones={test-zone2, test-zone3} 2021-03-12T02:40:06.841 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:06.887 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:06.950 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-41 zones={test-zone2, test-zone3} 2021-03-12T02:40:06.950 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-42 zones={test-zone2, test-zone1} 2021-03-12T02:40:06.960 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:07.011 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:07.073 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-42 zones={test-zone2, test-zone1} 2021-03-12T02:40:07.074 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-42 zones={test-zone3, test-zone1} 2021-03-12T02:40:07.083 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:07.135 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:07.198 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-42 zones={test-zone3, test-zone1} 2021-03-12T02:40:07.199 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-42 zones={test-zone1, test-zone2} 2021-03-12T02:40:07.208 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:07.254 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:07.316 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-42 zones={test-zone1, test-zone2} 2021-03-12T02:40:07.317 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-42 zones={test-zone3, test-zone2} 2021-03-12T02:40:07.327 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:07.375 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:07.440 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-42 zones={test-zone3, test-zone2} 2021-03-12T02:40:07.440 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-42 zones={test-zone1, test-zone3} 2021-03-12T02:40:07.452 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:07.498 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:07.566 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-42 zones={test-zone1, test-zone3} 2021-03-12T02:40:07.566 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-42 zones={test-zone2, test-zone3} 2021-03-12T02:40:07.576 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:07.627 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:07.689 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-42 zones={test-zone2, test-zone3} 2021-03-12T02:40:07.690 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-43 zones={test-zone2, test-zone1} 2021-03-12T02:40:07.699 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:07.751 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:07.814 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-43 zones={test-zone2, test-zone1} 2021-03-12T02:40:07.815 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-43 zones={test-zone3, test-zone1} 2021-03-12T02:40:07.825 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:07.871 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:07.938 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-43 zones={test-zone3, test-zone1} 2021-03-12T02:40:07.939 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-43 zones={test-zone1, test-zone2} 2021-03-12T02:40:07.953 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:08.007 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:08.072 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-43 zones={test-zone1, test-zone2} 2021-03-12T02:40:08.073 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-43 zones={test-zone3, test-zone2} 2021-03-12T02:40:08.084 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:08.131 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:08.199 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-43 zones={test-zone3, test-zone2} 2021-03-12T02:40:08.200 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-43 zones={test-zone1, test-zone3} 2021-03-12T02:40:08.214 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:08.266 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:08.331 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-43 zones={test-zone1, test-zone3} 2021-03-12T02:40:08.331 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-43 zones={test-zone2, test-zone3} 2021-03-12T02:40:08.342 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:08.390 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:08.455 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-43 zones={test-zone2, test-zone3} 2021-03-12T02:40:08.456 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-44 zones={test-zone2, test-zone1} 2021-03-12T02:40:08.465 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:08.514 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:08.582 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-44 zones={test-zone2, test-zone1} 2021-03-12T02:40:08.582 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-44 zones={test-zone3, test-zone1} 2021-03-12T02:40:08.592 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:08.642 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:08.706 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-44 zones={test-zone3, test-zone1} 2021-03-12T02:40:08.706 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-44 zones={test-zone1, test-zone2} 2021-03-12T02:40:08.716 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:08.763 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:08.824 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-44 zones={test-zone1, test-zone2} 2021-03-12T02:40:08.824 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-44 zones={test-zone3, test-zone2} 2021-03-12T02:40:08.833 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:08.883 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:08.947 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-44 zones={test-zone3, test-zone2} 2021-03-12T02:40:08.948 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-44 zones={test-zone1, test-zone3} 2021-03-12T02:40:08.960 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:09.010 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:09.073 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-44 zones={test-zone1, test-zone3} 2021-03-12T02:40:09.074 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-44 zones={test-zone2, test-zone3} 2021-03-12T02:40:09.084 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:09.135 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:09.197 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-44 zones={test-zone2, test-zone3} 2021-03-12T02:40:09.198 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-45 zones={test-zone2, test-zone1} 2021-03-12T02:40:09.209 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:09.259 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:09.323 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-45 zones={test-zone2, test-zone1} 2021-03-12T02:40:09.324 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-45 zones={test-zone3, test-zone1} 2021-03-12T02:40:09.332 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:09.379 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:09.446 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-45 zones={test-zone3, test-zone1} 2021-03-12T02:40:09.447 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-45 zones={test-zone1, test-zone2} 2021-03-12T02:40:09.458 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:09.507 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:09.568 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-45 zones={test-zone1, test-zone2} 2021-03-12T02:40:09.569 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-45 zones={test-zone3, test-zone2} 2021-03-12T02:40:09.579 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:09.626 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:09.688 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-45 zones={test-zone3, test-zone2} 2021-03-12T02:40:09.688 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-45 zones={test-zone1, test-zone3} 2021-03-12T02:40:09.700 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:09.747 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:09.809 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-45 zones={test-zone1, test-zone3} 2021-03-12T02:40:09.809 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-45 zones={test-zone2, test-zone3} 2021-03-12T02:40:09.818 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:40:09.867 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:40:09.930 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-45 zones={test-zone2, test-zone3} 2021-03-12T02:40:09.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0cf3ee11-29e6-4776-941b-50bda62adf82', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:09.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0cf3ee11-29e6-4776-941b-50bda62adf82', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:09.932 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0cf3ee11-29e6-4776-941b-50bda62adf82 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:09.972 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:09.973 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:10.206 INFO:teuthology.orchestra.run.smithi077.stdout:[{"id":"1_1615516156.388233_96.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-12T02:29:16.388233974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YL3_4pFMNCff-5Idsb0xw0Y","ver":1},"status":{"status":"write"}}},{"id":"1_1615516156.395861_98.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-12T02:29:16.395861436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YL3_4pFMNCff-5Idsb0xw0Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.517635_58.1","section":"bucket.instance","name":"bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19","timestamp":"2021-03-12T02:31:57.517635735Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jcLKJHBt6Ev7gDksP2Slh3b","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.523069_59.1","section":"bucket.instance","name":"bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19","timestamp":"2021-03-12T02:31:57.523069746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jcLKJHBt6Ev7gDksP2Slh3b","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.525015_60.1","section":"bucket","name":"bbucnh-10","timestamp":"2021-03-12T02:31:57.525015146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6lPftr6jYyKA9E3-g-_Jn6a","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.529693_61.1","section":"bucket","name":"bbucnh-10","timestamp":"2021-03-12T02:31:57.529693741Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6lPftr6jYyKA9E3-g-_Jn6a","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.924299_62.1","section":"bucket.instance","name":"bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20","timestamp":"2021-03-12T02:31:57.924299109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J0Dn8QzeuUr9s9ivyu7dsOD","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.930629_63.1","section":"bucket.instance","name":"bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20","timestamp":"2021-03-12T02:31:57.930629043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J0Dn8QzeuUr9s9ivyu7dsOD","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.932410_64.1","section":"bucket","name":"bbucnh-11","timestamp":"2021-03-12T02:31:57.932410309Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zofeXVzfWAE58gjk_kRUwx9","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.936714_65.1","section":"bucket","name":"bbucnh-11","timestamp":"2021-03-12T02:31:57.936714991Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zofeXVzfWAE58gjk_kRUwx9","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.973905_66.1","section":"bucket.instance","name":"bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21","timestamp":"2021-03-12T02:31:57.973905477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_25ydPCfdlzXA1KPDlkW_pFL","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.978105_67.1","section":"bucket.instance","name":"bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21","timestamp":"2021-03-12T02:31:57.978105914Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_25ydPCfdlzXA1KPDlkW_pFL","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.979768_68.1","section":"bucket","name":"bbucnh-12","timestamp":"2021-03-12T02:31:57.979768274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dw8z_OfXq9g-1vCkmj9nOKx","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.983732_69.1","section":"bucket","name":"bbucnh-12","timestamp":"2021-03-12T02:31:57.983732995Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dw8z_OfXq9g-1vCkmj9nOKx","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516335.993721_86.1","section":"bucket.instance","name":"bbucnh-13:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22","timestamp":"2021-03-12T02:32:15.993721453Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iFsto3hGG7H5MJRUgU2zSGi","ver":1},"status":{"status":"write"}}},{"id":"1_1615516335.999815_87.1","section":"bucket.instance","name":"bbucnh-13:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22","timestamp":"2021-03-12T02:32:15.999815759Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iFsto3hGG7H5MJRUgU2zSGi","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.001721_88.1","section":"bucket","name":"bbucnh-13","timestamp":"2021-03-12T02:32:16.001721436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dy_6thslFFy5DEBDJ7AdBBZ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.006149_89.1","section":"bucket","name":"bbucnh-13","timestamp":"2021-03-12T02:32:16.006149764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dy_6thslFFy5DEBDJ7AdBBZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.023548_90.1","section":"bucket.instance","name":"bbucnh-14:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23","timestamp":"2021-03-12T02:32:16.023548341Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UidaAgtGQcz4suSP3NCWLHf","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.028425_91.1","section":"bucket.instance","name":"bbucnh-14:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23","timestamp":"2021-03-12T02:32:16.028425620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UidaAgtGQcz4suSP3NCWLHf","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.030253_92.1","section":"bucket","name":"bbucnh-14","timestamp":"2021-03-12T02:32:16.030253029Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8u_rE6TyERk7cInsKIFxC_W","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.034867_93.1","section":"bucket","name":"bbucnh-14","timestamp":"2021-03-12T02:32:16.034867278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8u_rE6TyERk7cInsKIFxC_W","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.073203_94.1","section":"bucket.instance","name":"bbucnh-15:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24","timestamp":"2021-03-12T02:32:16.073203702Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fPI6vD6iqij_2lOipNCA5Pj","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.078448_95.1","section":"bucket.instance","name":"bbucnh-15:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24","timestamp":"2021-03-12T02:32:16.078448093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fPI6vD6iqij_2lOipNCA5Pj","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.079985_96.1","section":"bucket","name":"bbucnh-15","timestamp":"2021-03-12T02:32:16.079985141Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZURiB1MDtruSPPRLjEHfFQn","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.084220_97.1","section":"bucket","name":"bbucnh-15","timestamp":"2021-03-12T02:32:16.084220471Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZURiB1MDtruSPPRLjEHfFQn","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.698485_109.1","section":"bucket.instance","name":"bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25","timestamp":"2021-03-12T02:33:02.698485717Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.704669_110.1","section":"bucket.instance","name":"bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25","timestamp":"2021-03-12T02:33:02.704669333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.706515_111.1","section":"bucket","name":"bbucnh-16","timestamp":"2021-03-12T02:33:02.706515387Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3i1dwkEn1BGUdDnt5jjMXCT","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.713003_112.1","section":"bucket","name":"bbucnh-16","timestamp":"2021-03-12T02:33:02.713003405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3i1dwkEn1BGUdDnt5jjMXCT","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.737854_113.1","section":"bucket.instance","name":"bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26","timestamp":"2021-03-12T02:33:02.737854190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.743984_114.1","section":"bucket.instance","name":"bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26","timestamp":"2021-03-12T02:33:02.743984571Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.745895_115.1","section":"bucket","name":"bbucnh-17","timestamp":"2021-03-12T02:33:02.745895289Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cPb986KctBPZNxu1j2usRVk","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.751955_116.1","section":"bucket","name":"bbucnh-17","timestamp":"2021-03-12T02:33:02.751955799Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cPb986KctBPZNxu1j2usRVk","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.793972_117.1","section":"bucket.instance","name":"bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27","timestamp":"2021-03-12T02:33:02.793972919Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.799476_118.1","section":"bucket.instance","name":"bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27","timestamp":"2021-03-12T02:33:02.799476294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.801233_119.1","section":"bucket","name":"bbucnh-18","timestamp":"2021-03-12T02:33:02.801233293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bE0Z6UtgXyyVGQt7bxmNWss","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.805888_120.1","section":"bucket","name":"bbucnh-18","timestamp":"2021-03-12T02:33:02.805888319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bE0Z6UtgXyyVGQt7bxmNWss","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.833909_121.1","section":"bucket.instance","name":"bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25","timestamp":"2021-03-12T02:33:02.833909432Z","data":{"read_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":1},"write_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":2},"status":{"status":"write"}}},{"id":"1_1615516382.839228_122.1","section":"bucket.instance","name":"bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25","timestamp":"2021-03-12T02:33:02.839228126Z","data":{"read_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":1},"write_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516382.844254_123.1","section":"bucket.instance","name":"bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26","timestamp":"2021-03-12T02:33:02.844254956Z","data":{"read_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":1},"write_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":2},"status":{"status":"write"}}},{"id":"1_1615516382.850139_124.1","section":"bucket.instance","name":"bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26","timestamp":"2021-03-12T02:33:02.850139352Z","data":{"read_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":1},"write_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516382.860106_125.1","section":"bucket.instance","name":"bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27","timestamp":"2021-03-12T02:33:02.860106932Z","data":{"read_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":1},"write_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":2},"status":{"status":"write"}}},{"id":"1_1615516382.866387_126.1","section":"bucket.instance","name":"bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27","timestamp":"2021-03-12T02:33:02.866387331Z","data":{"read_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":1},"write_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516477.625386_179.1","section":"bucket.instance","name":"bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28","timestamp":"2021-03-12T02:34:37.625386901Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":1},"status":{"status":"write"}}},{"id":"1_1615516477.630575_180.1","section":"bucket.instance","name":"bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28","timestamp":"2021-03-12T02:34:37.630575238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516477.632196_181.1","section":"bucket","name":"bbucnh-19","timestamp":"2021-03-12T02:34:37.632196980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nPMLF28l579DSvKw8ZKyOBr","ver":1},"status":{"status":"write"}}},{"id":"1_1615516477.636638_182.1","section":"bucket","name":"bbucnh-19","timestamp":"2021-03-12T02:34:37.636638396Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nPMLF28l579DSvKw8ZKyOBr","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516477.641759_183.1","section":"bucket.instance","name":"bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28","timestamp":"2021-03-12T02:34:37.641759734Z","data":{"read_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":1},"write_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":2},"status":{"status":"write"}}},{"id":"1_1615516477.646256_184.1","section":"bucket.instance","name":"bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28","timestamp":"2021-03-12T02:34:37.646256599Z","data":{"read_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":1},"write_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516542.961983_188.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:35:42.961983629Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":1},"status":{"status":"write"}}},{"id":"1_1615516542.966262_189.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:35:42.966262443Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516542.968145_190.1","section":"bucket","name":"bbucnh-20","timestamp":"2021-03-12T02:35:42.968145462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jmYhKySMPNTjtHT_ULsBbuI","ver":1},"status":{"status":"write"}}},{"id":"1_1615516542.972050_191.1","section":"bucket","name":"bbucnh-20","timestamp":"2021-03-12T02:35:42.972050294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jmYhKySMPNTjtHT_ULsBbuI","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516560.741881_192.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:36:00.741881668Z","data":{"read_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":1},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":2},"status":{"status":"write"}}},{"id":"1_1615516560.750253_193.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:36:00.750253702Z","data":{"read_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":1},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516580.510392_194.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:36:20.510392140Z","data":{"read_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":2},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":3},"status":{"status":"write"}}},{"id":"1_1615516580.518628_195.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:36:20.518628079Z","data":{"read_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":2},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":3},"status":{"status":"complete"}}},{"id":"1_1615516601.288975_196.1","section":"bucket.instance","name":"bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30","timestamp":"2021-03-12T02:36:41.288975015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.293430_197.1","section":"bucket.instance","name":"bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30","timestamp":"2021-03-12T02:36:41.293430632Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.295149_198.1","section":"bucket","name":"bbucnh-21","timestamp":"2021-03-12T02:36:41.295149172Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__zd-Z1R4RWVmQc9qnWCZJ9Z","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.301563_199.1","section":"bucket","name":"bbucnh-21","timestamp":"2021-03-12T02:36:41.301563274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__zd-Z1R4RWVmQc9qnWCZJ9Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.326503_200.1","section":"bucket.instance","name":"bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31","timestamp":"2021-03-12T02:36:41.326503965Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.331867_201.1","section":"bucket.instance","name":"bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31","timestamp":"2021-03-12T02:36:41.331867729Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.333775_202.1","section":"bucket","name":"bbucnh-22","timestamp":"2021-03-12T02:36:41.333775173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lRVZbkb3M5VDTFzWzb2Ao_n","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.338365_203.1","section":"bucket","name":"bbucnh-22","timestamp":"2021-03-12T02:36:41.338365599Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lRVZbkb3M5VDTFzWzb2Ao_n","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.375262_204.1","section":"bucket.instance","name":"bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32","timestamp":"2021-03-12T02:36:41.375262696Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.380184_205.1","section":"bucket.instance","name":"bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32","timestamp":"2021-03-12T02:36:41.380184001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.382095_206.1","section":"bucket","name":"bbucnh-23","timestamp":"2021-03-12T02:36:41.382095207Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ox0KlvX6xNF4jdqzrljR_Yz","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.386448_207.1","section":"bucket","name":"bbucnh-23","timestamp":"2021-03-12T02:36:41.386448784Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ox0KlvX6xNF4jdqzrljR_Yz","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.406400_208.1","section":"bucket.instance","name":"bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30","timestamp":"2021-03-12T02:36:41.406400290Z","data":{"read_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":1},"write_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":2},"status":{"status":"write"}}},{"id":"1_1615516601.411130_209.1","section":"bucket.instance","name":"bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30","timestamp":"2021-03-12T02:36:41.411130339Z","data":{"read_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":1},"write_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516601.415296_210.1","section":"bucket.instance","name":"bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31","timestamp":"2021-03-12T02:36:41.415296943Z","data":{"read_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":1},"write_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":2},"status":{"status":"write"}}},{"id":"1_1615516601.420232_211.1","section":"bucket.instance","name":"bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31","timestamp":"2021-03-12T02:36:41.420232785Z","data":{"read_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":1},"write_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516601.429424_212.1","section":"bucket.instance","name":"bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32","timestamp":"2021-03-12T02:36:41.429424011Z","data":{"read_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":1},"write_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":2},"status":{"status":"write"}}},{"id":"1_1615516601.435061_213.1","section":"bucket.instance","name":"bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32","timestamp":"2021-03-12T02:36:41.435061595Z","data":{"read_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":1},"write_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516623.905134_226.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:03.905134670Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":1},"status":{"status":"write"}}},{"id":"1_1615516623.909582_227.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:03.909582485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516623.911054_228.1","section":"bucket","name":"bbucnh-24","timestamp":"2021-03-12T02:37:03.911054290Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kAh_Dud7mDE7666DS44bDVH","ver":1},"status":{"status":"write"}}},{"id":"1_1615516623.915168_229.1","section":"bucket","name":"bbucnh-24","timestamp":"2021-03-12T02:37:03.915168538Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kAh_Dud7mDE7666DS44bDVH","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516623.938779_230.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:03.938779082Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":1},"status":{"status":"write"}}},{"id":"1_1615516623.944057_231.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:03.944057002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516623.945869_232.1","section":"bucket","name":"bbucnh-25","timestamp":"2021-03-12T02:37:03.945869854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FVWFGEedKW_N26l2XXHS__P","ver":1},"status":{"status":"write"}}},{"id":"1_1615516623.952222_233.1","section":"bucket","name":"bbucnh-25","timestamp":"2021-03-12T02:37:03.952222510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FVWFGEedKW_N26l2XXHS__P","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516623.998369_234.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:03.998369746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":1},"status":{"status":"write"}}},{"id":"1_1615516624.003251_235.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.003251527Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516624.004804_236.1","section":"bucket","name":"bbucnh-26","timestamp":"2021-03-12T02:37:04.004804195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pZWbNId4gDq01c4QPLoSkT3","ver":1},"status":{"status":"write"}}},{"id":"1_1615516624.009867_237.1","section":"bucket","name":"bbucnh-26","timestamp":"2021-03-12T02:37:04.009867850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pZWbNId4gDq01c4QPLoSkT3","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516624.034129_238.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:04.034129873Z","data":{"read_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":1},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":2},"status":{"status":"write"}}},{"id":"1_1615516624.040808_239.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:04.040808023Z","data":{"read_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":1},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516624.044134_240.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:04.044134611Z","data":{"read_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":2},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":3},"status":{"status":"write"}}},{"id":"1_1615516624.051708_241.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:04.051708030Z","data":{"read_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":2},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":3},"status":{"status":"complete"}}},{"id":"1_1615516624.056887_242.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:04.056887996Z","data":{"read_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":1},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":2},"status":{"status":"write"}}},{"id":"1_1615516624.063220_243.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:04.063220593Z","data":{"read_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":1},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516624.073512_244.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:04.073512038Z","data":{"read_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":2},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":3},"status":{"status":"write"}}},{"id":"1_1615516624.080300_245.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:04.080300997Z","data":{"read_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":2},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":3},"status":{"status":"complete"}}},{"id":"1_1615516624.092046_246.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.092046934Z","data":{"read_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":1},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":2},"status":{"status":"write"}}},{"id":"1_1615516624.098396_247.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.098396048Z","data":{"read_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":1},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516624.109918_248.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.109918336Z","data":{"read_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":2},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":3},"status":{"status":"write"}}},{"id":"1_1615516624.119886_249.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.119886963Z","data":{"read_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":2},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":3},"status":{"status":"complete"}}},{"id":"1_1615516648.953955_260.1","section":"bucket.instance","name":"bbucnh-27:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.36","timestamp":"2021-03-12T02:37:28.953955696Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516648.958378_261.1","section":"bucket.instance","name":"bbucnh-27:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.36","timestamp":"2021-03-12T02:37:28.958378293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516648.959965_262.1","section":"bucket","name":"bbucnh-27","timestamp":"2021-03-12T02:37:28.959965688Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FmfQYONhZDXgO1uw7EYs9_S","ver":1},"status":{"status":"write"}}},{"id":"1_1615516648.963902_263.1","section":"bucket","name":"bbucnh-27","timestamp":"2021-03-12T02:37:28.963902711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FmfQYONhZDXgO1uw7EYs9_S","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516648.987286_264.1","section":"bucket.instance","name":"bbucnh-28:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37","timestamp":"2021-03-12T02:37:28.987286166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":1},"status":{"status":"write"}}},{"id":"1_1615516648.993373_265.1","section":"bucket.instance","name":"bbucnh-28:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37","timestamp":"2021-03-12T02:37:28.993373542Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516648.995310_266.1","section":"bucket","name":"bbucnh-28","timestamp":"2021-03-12T02:37:28.995310434Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2gVXe3dHFNHJj_5mBzi7BFQ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.000050_267.1","section":"bucket","name":"bbucnh-28","timestamp":"2021-03-12T02:37:29.000050799Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2gVXe3dHFNHJj_5mBzi7BFQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.040216_268.1","section":"bucket.instance","name":"bbucnh-29:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38","timestamp":"2021-03-12T02:37:29.040216016Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.045685_269.1","section":"bucket.instance","name":"bbucnh-29:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38","timestamp":"2021-03-12T02:37:29.045685128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.047750_270.1","section":"bucket","name":"bbucnh-29","timestamp":"2021-03-12T02:37:29.047750134Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NzPM--eEDeYlpvhbh0MdeK4","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.052200_271.1","section":"bucket","name":"bbucnh-29","timestamp":"2021-03-12T02:37:29.052200209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NzPM--eEDeYlpvhbh0MdeK4","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.075303_272.1","section":"bucket.instance","name":"bbucnh-27:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.36","timestamp":"2021-03-12T02:37:29.075303587Z","data":{"read_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":1},"write_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.080878_273.1","section":"bucket.instance","name":"bbucnh-27:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.36","timestamp":"2021-03-12T02:37:29.080878607Z","data":{"read_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":1},"write_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.129020_274.1","section":"bucket.instance","name":"bbucnh-28:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37","timestamp":"2021-03-12T02:37:29.129020143Z","data":{"read_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":1},"write_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.137207_275.1","section":"bucket.instance","name":"bbucnh-28:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37","timestamp":"2021-03-12T02:37:29.137207384Z","data":{"read_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":1},"write_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.196723_276.1","section":"bucket.instance","name":"bbucnh-29:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38","timestamp":"2021-03-12T02:37:29.196723737Z","data":{"read_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":1},"write_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.204941_277.1","section":"bucket.instance","name":"bbucnh-29:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38","timestamp":"2021-03-12T02:37:29.204941290Z","data":{"read_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":1},"write_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.273026_278.1","section":"bucket.instance","name":"bbucnh-30:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.39","timestamp":"2021-03-12T02:37:29.273026550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.278145_279.1","section":"bucket.instance","name":"bbucnh-30:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.39","timestamp":"2021-03-12T02:37:29.278145289Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.279975_280.1","section":"bucket","name":"bbucnh-30","timestamp":"2021-03-12T02:37:29.279975374Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ktkorz1HBIgiOiSGUck37Fi","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.284871_281.1","section":"bucket","name":"bbucnh-30","timestamp":"2021-03-12T02:37:29.284871749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ktkorz1HBIgiOiSGUck37Fi","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.312142_282.1","section":"bucket.instance","name":"bbucnh-31:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40","timestamp":"2021-03-12T02:37:29.312142460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.317109_283.1","section":"bucket.instance","name":"bbucnh-31:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40","timestamp":"2021-03-12T02:37:29.317109530Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.318899_284.1","section":"bucket","name":"bbucnh-31","timestamp":"2021-03-12T02:37:29.318899628Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RVIUeIv4NvsRyQ3Z9VfRls-","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.323537_285.1","section":"bucket","name":"bbucnh-31","timestamp":"2021-03-12T02:37:29.323537422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RVIUeIv4NvsRyQ3Z9VfRls-","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.364204_286.1","section":"bucket.instance","name":"bbucnh-32:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41","timestamp":"2021-03-12T02:37:29.364204861Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.369006_287.1","section":"bucket.instance","name":"bbucnh-32:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41","timestamp":"2021-03-12T02:37:29.369006270Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.370821_288.1","section":"bucket","name":"bbucnh-32","timestamp":"2021-03-12T02:37:29.370821226Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uNNdkr5c4kYGRpyB0KL4L6X","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.374864_289.1","section":"bucket","name":"bbucnh-32","timestamp":"2021-03-12T02:37:29.374864824Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uNNdkr5c4kYGRpyB0KL4L6X","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.400473_291.1","section":"bucket.instance","name":"bbucnh-30:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.39","timestamp":"2021-03-12T02:37:29.400473777Z","data":{"read_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":1},"write_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.405529_292.1","section":"bucket.instance","name":"bbucnh-30:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.39","timestamp":"2021-03-12T02:37:29.405529038Z","data":{"read_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":1},"write_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.454396_293.1","section":"bucket.instance","name":"bbucnh-31:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40","timestamp":"2021-03-12T02:37:29.454396735Z","data":{"read_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":1},"write_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.462159_294.1","section":"bucket.instance","name":"bbucnh-31:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40","timestamp":"2021-03-12T02:37:29.462159544Z","data":{"read_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":1},"write_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.526776_295.1","section":"bucket.instance","name":"bbucnh-32:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41","timestamp":"2021-03-12T02:37:29.526776297Z","data":{"read_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":1},"write_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.534441_296.1","section":"bucket.instance","name":"bbucnh-32:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41","timestamp":"2021-03-12T02:37:29.534441885Z","data":{"read_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":1},"write_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.609051_298.1","section":"bucket.instance","name":"bbucnh-33:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.42","timestamp":"2021-03-12T02:37:29.609051947Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.613737_299.1","section":"bucket.instance","name":"bbucnh-33:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.42","timestamp":"2021-03-12T02:37:29.613737442Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.615385_300.1","section":"bucket","name":"bbucnh-33","timestamp":"2021-03-12T02:37:29.615385271Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_15Kd9p4ie5l3fyoCESSGjmB","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.619356_301.1","section":"bucket","name":"bbucnh-33","timestamp":"2021-03-12T02:37:29.619356209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_15Kd9p4ie5l3fyoCESSGjmB","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.644293_302.1","section":"bucket.instance","name":"bbucnh-34:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43","timestamp":"2021-03-12T02:37:29.644293956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.649652_303.1","section":"bucket.instance","name":"bbucnh-34:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43","timestamp":"2021-03-12T02:37:29.649652088Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.651633_304.1","section":"bucket","name":"bbucnh-34","timestamp":"2021-03-12T02:37:29.651633615Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KxY1ghborztF7ijmoRBmfYR","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.656854_305.1","section":"bucket","name":"bbucnh-34","timestamp":"2021-03-12T02:37:29.656854451Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KxY1ghborztF7ijmoRBmfYR","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.700347_306.1","section":"bucket.instance","name":"bbucnh-35:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44","timestamp":"2021-03-12T02:37:29.700347440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.705720_307.1","section":"bucket.instance","name":"bbucnh-35:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44","timestamp":"2021-03-12T02:37:29.705720561Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.707370_308.1","section":"bucket","name":"bbucnh-35","timestamp":"2021-03-12T02:37:29.707370179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tJebU3psH_SbzeVFZfWinUh","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.711408_309.1","section":"bucket","name":"bbucnh-35","timestamp":"2021-03-12T02:37:29.711408051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tJebU3psH_SbzeVFZfWinUh","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.734891_310.1","section":"bucket.instance","name":"bbucnh-33:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.42","timestamp":"2021-03-12T02:37:29.734891456Z","data":{"read_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":1},"write_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.739920_311.1","section":"bucket.instance","name":"bbucnh-33:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.42","timestamp":"2021-03-12T02:37:29.739920185Z","data":{"read_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":1},"write_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516650.789737_312.1","section":"bucket.instance","name":"bbucnh-34:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43","timestamp":"2021-03-12T02:37:30.789737190Z","data":{"read_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":1},"write_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":2},"status":{"status":"write"}}},{"id":"1_1615516650.798206_313.1","section":"bucket.instance","name":"bbucnh-34:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43","timestamp":"2021-03-12T02:37:30.798206828Z","data":{"read_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":1},"write_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516652.335363_314.1","section":"bucket.instance","name":"bbucnh-35:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44","timestamp":"2021-03-12T02:37:32.335363586Z","data":{"read_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":1},"write_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":2},"status":{"status":"write"}}},{"id":"1_1615516652.344811_315.1","section":"bucket.instance","name":"bbucnh-35:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44","timestamp":"2021-03-12T02:37:32.344811119Z","data":{"read_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":1},"write_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516653.069940_316.1","section":"bucket.instance","name":"bbucnh-36:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45","timestamp":"2021-03-12T02:37:33.069940667Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VuDCjriGUmI_jGsCkSwyMPM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516653.075083_317.1","section":"bucket.instance","name":"bbucnh-36:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45","timestamp":"2021-03-12T02:37:33.075083097Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VuDCjriGUmI_jGsCkSwyMPM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516653.076710_318.1","section":"bucket","name":"bbucnh-36","timestamp":"2021-03-12T02:37:33.076710866Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6lGSmQvlrfayDP125PAmHZF","ver":1},"status":{"status":"write"}}},{"id":"1_1615516653.081538_319.1","section":"bucket","name":"bbucnh-36","timestamp":"2021-03-12T02:37:33.081538100Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6lGSmQvlrfayDP125PAmHZF","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516653.664443_320.1","section":"bucket.instance","name":"bbucnh-37:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46","timestamp":"2021-03-12T02:37:33.664443824Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HmfgDDWFN0bOGRAMO2ByNk7","ver":1},"status":{"status":"write"}}},{"id":"1_1615516653.669629_321.1","section":"bucket.instance","name":"bbucnh-37:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46","timestamp":"2021-03-12T02:37:33.669629721Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HmfgDDWFN0bOGRAMO2ByNk7","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516653.671304_322.1","section":"bucket","name":"bbucnh-37","timestamp":"2021-03-12T02:37:33.671304239Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J6fc8NV-QkiOt2H3vUtcdSS","ver":1},"status":{"status":"write"}}},{"id":"1_1615516653.675804_323.1","section":"bucket","name":"bbucnh-37","timestamp":"2021-03-12T02:37:33.675804856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J6fc8NV-QkiOt2H3vUtcdSS","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516653.998533_324.1","section":"bucket.instance","name":"bbucnh-38:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47","timestamp":"2021-03-12T02:37:33.998533211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DXFrPBq7cR1YlfkiP9rTC_h","ver":1},"status":{"status":"write"}}},{"id":"1_1615516654.002986_325.1","section":"bucket.instance","name":"bbucnh-38:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47","timestamp":"2021-03-12T02:37:34.002986261Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DXFrPBq7cR1YlfkiP9rTC_h","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516654.004526_326.1","section":"bucket","name":"bbucnh-38","timestamp":"2021-03-12T02:37:34.004526175Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B2U5VzBNff39d-uvhTwPTVA","ver":1},"status":{"status":"write"}}},{"id":"1_1615516654.008644_327.1","section":"bucket","name":"bbucnh-38","timestamp":"2021-03-12T02:37:34.008644344Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B2U5VzBNff39d-uvhTwPTVA","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.472251_328.1","section":"bucket.instance","name":"bbucnh-39:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.48","timestamp":"2021-03-12T02:37:36.472251903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxwXPsBw5MBOuLy2DL0RzSC","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.477307_329.1","section":"bucket.instance","name":"bbucnh-39:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.48","timestamp":"2021-03-12T02:37:36.477307238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxwXPsBw5MBOuLy2DL0RzSC","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.479321_330.1","section":"bucket","name":"bbucnh-39","timestamp":"2021-03-12T02:37:36.479321851Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7RmOoO6yS0MAc_ELYlYq8yN","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.483505_331.1","section":"bucket","name":"bbucnh-39","timestamp":"2021-03-12T02:37:36.483505123Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7RmOoO6yS0MAc_ELYlYq8yN","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.511042_332.1","section":"bucket.instance","name":"bbucnh-40:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.49","timestamp":"2021-03-12T02:37:36.511042128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q1cAgCTgGfyj4SRitg3YHYj","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.516504_333.1","section":"bucket.instance","name":"bbucnh-40:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.49","timestamp":"2021-03-12T02:37:36.516504986Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q1cAgCTgGfyj4SRitg3YHYj","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.518471_334.1","section":"bucket","name":"bbucnh-40","timestamp":"2021-03-12T02:37:36.518471862Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vmcO4AQPJh3F7zPA8Q1MDcO","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.522955_335.1","section":"bucket","name":"bbucnh-40","timestamp":"2021-03-12T02:37:36.522955683Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vmcO4AQPJh3F7zPA8Q1MDcO","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.569936_336.1","section":"bucket.instance","name":"bbucnh-41:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.50","timestamp":"2021-03-12T02:37:36.569936921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V9oej-UfPhBNN29T2LbubWl","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.574506_337.1","section":"bucket.instance","name":"bbucnh-41:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.50","timestamp":"2021-03-12T02:37:36.574506107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V9oej-UfPhBNN29T2LbubWl","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.576240_338.1","section":"bucket","name":"bbucnh-41","timestamp":"2021-03-12T02:37:36.576240547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TmSKoQq5CYyZMuTTOWMF8jB","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.580426_339.1","section":"bucket","name":"bbucnh-41","timestamp":"2021-03-12T02:37:36.580426570Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TmSKoQq5CYyZMuTTOWMF8jB","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516277.246267_81.1","section":"bucket.instance","name":"bbucnh-1:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.1","timestamp":"2021-03-12T02:31:17.246267120Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qfxaS5zdrJis58Zg6Obg75g","ver":1},"status":{"status":"write"}}},{"id":"1_1615516277.255262_85.1","section":"bucket.instance","name":"bbucnh-1:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.1","timestamp":"2021-03-12T02:31:17.255262459Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qfxaS5zdrJis58Zg6Obg75g","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516277.256664_86.1","section":"bucket","name":"bbucnh-1","timestamp":"2021-03-12T02:31:17.256664887Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SJQ8Ox0nk_KuM-tokbc3CHJ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516277.260851_87.1","section":"bucket","name":"bbucnh-1","timestamp":"2021-03-12T02:31:17.260851736Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SJQ8Ox0nk_KuM-tokbc3CHJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516277.279301_88.1","section":"bucket.instance","name":"bbucnh-2:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.2","timestamp":"2021-03-12T02:31:17.279301358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OdmYkD6xh5KguUnb-mdZ1aM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516277.284169_89.1","section":"bucket.instance","name":"bbucnh-2:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.2","timestamp":"2021-03-12T02:31:17.284169621Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OdmYkD6xh5KguUnb-mdZ1aM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516277.285818_90.1","section":"bucket","name":"bbucnh-2","timestamp":"2021-03-12T02:31:17.285818007Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U61zBQN8Y4-IaOqXqUfnFDK","ver":1},"status":{"status":"write"}}},{"id":"1_1615516277.289902_91.1","section":"bucket","name":"bbucnh-2","timestamp":"2021-03-12T02:31:17.289902101Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U61zBQN8Y4-IaOqXqUfnFDK","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516280.582981_92.1","section":"bucket.instance","name":"bbucnh-3:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.3","timestamp":"2021-03-12T02:31:20.582981563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WmExOXe_DXwUuw8ZWwStmCG","ver":1},"status":{"status":"write"}}},{"id":"1_1615516280.587282_93.1","section":"bucket.instance","name":"bbucnh-3:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.3","timestamp":"2021-03-12T02:31:20.587282731Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WmExOXe_DXwUuw8ZWwStmCG","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516280.588897_94.1","section":"bucket","name":"bbucnh-3","timestamp":"2021-03-12T02:31:20.588897050Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vTlspttjgtqvLuUNPCk_X1Z","ver":1},"status":{"status":"write"}}},{"id":"1_1615516280.593132_95.1","section":"bucket","name":"bbucnh-3","timestamp":"2021-03-12T02:31:20.593132328Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vTlspttjgtqvLuUNPCk_X1Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516289.394488_96.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.4","timestamp":"2021-03-12T02:31:29.394488830Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P9XE_bWiNIw3vvd5uTEfxr3","ver":1},"status":{"status":"write"}}},{"id":"1_1615516289.399511_98.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.4","timestamp":"2021-03-12T02:31:29.399511189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P9XE_bWiNIw3vvd5uTEfxr3","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516289.400904_99.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:29.400904315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HJA5lWRMWyY0IaqyBfFi1N4","ver":1},"status":{"status":"write"}}},{"id":"1_1615516289.404420_100.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:29.404420265Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HJA5lWRMWyY0IaqyBfFi1N4","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516289.903936_101.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.5","timestamp":"2021-03-12T02:31:29.903936280Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1ecwCnmWKxBMRrpYQmvXMe-","ver":1},"status":{"status":"write"}}},{"id":"1_1615516289.908750_102.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.5","timestamp":"2021-03-12T02:31:29.908750593Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1ecwCnmWKxBMRrpYQmvXMe-","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516289.910808_103.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:29.910808743Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nZiOpvQgZemMdcUiOXIn1VT","ver":1},"status":{"status":"write"}}},{"id":"1_1615516289.916752_104.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:29.916752691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nZiOpvQgZemMdcUiOXIn1VT","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516290.449434_105.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.6","timestamp":"2021-03-12T02:31:30.449434628Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rHxZdUR046Wgm3Zafw0Xi8l","ver":1},"status":{"status":"write"}}},{"id":"1_1615516290.453654_106.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.6","timestamp":"2021-03-12T02:31:30.453654797Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rHxZdUR046Wgm3Zafw0Xi8l","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516290.455139_107.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:30.455139379Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sg78H-Zgxy2C6QwruYgc-Rg","ver":1},"status":{"status":"write"}}},{"id":"1_1615516290.459112_108.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:30.459112128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sg78H-Zgxy2C6QwruYgc-Rg","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.676123_109.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.7","timestamp":"2021-03-12T02:31:32.676123449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T4BoTd-m-PH42nJonMN7kZP","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.681307_111.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.7","timestamp":"2021-03-12T02:31:32.681307323Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T4BoTd-m-PH42nJonMN7kZP","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.683099_112.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:32.683099550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EBuzwkBweLN5rvCeTKt4REO","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.688466_113.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:32.688466865Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EBuzwkBweLN5rvCeTKt4REO","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.712314_114.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.8","timestamp":"2021-03-12T02:31:32.712314715Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c8czZW2QuWkq0wJE4l-Nvwz","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.716801_115.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.8","timestamp":"2021-03-12T02:31:32.716801906Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c8czZW2QuWkq0wJE4l-Nvwz","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.718398_116.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:32.718398180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yjAubSLjTLPIuEzKEWdFJD8","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.721131_117.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:32.721131522Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yjAubSLjTLPIuEzKEWdFJD8","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.742288_118.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.9","timestamp":"2021-03-12T02:31:32.742288024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cd_ZmHPW-yz7g9wMfu8z1OM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.746471_119.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.9","timestamp":"2021-03-12T02:31:32.746471019Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cd_ZmHPW-yz7g9wMfu8z1OM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.748067_120.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:32.748067528Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yyCMzqqMnUhRq-wn-e0Oe_1","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.752347_121.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:32.752347222Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yyCMzqqMnUhRq-wn-e0Oe_1","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.782913_122.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.10","timestamp":"2021-03-12T02:31:32.782913622Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cbSranN-_o_cTrR-FcWF_2Y","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.788130_123.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.10","timestamp":"2021-03-12T02:31:32.788130810Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cbSranN-_o_cTrR-FcWF_2Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.790158_124.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:32.790158153Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7PSv7RO8pR2twgmTFBWqx0N","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.795541_125.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:32.795541882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7PSv7RO8pR2twgmTFBWqx0N","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.844603_126.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.11","timestamp":"2021-03-12T02:31:32.844603751Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4_B8b59PwgOUx0apcPz4YE3","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.849848_127.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.11","timestamp":"2021-03-12T02:31:32.849848924Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4_B8b59PwgOUx0apcPz4YE3","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.851664_128.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:32.851664510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G_5ocbuuJ8e81M9Oey3KkUa","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.855275_129.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:32.855275037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G_5ocbuuJ8e81M9Oey3KkUa","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.947746_130.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.12","timestamp":"2021-03-12T02:31:32.947746750Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hLAW_-1vXKVBgOh8Z3yp4RR","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.952705_131.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.12","timestamp":"2021-03-12T02:31:32.952705876Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hLAW_-1vXKVBgOh8Z3yp4RR","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.954374_132.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:32.954374195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u-RVInQ17b5MGi2bJAMXC9c","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.957220_133.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:32.957220699Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u-RVInQ17b5MGi2bJAMXC9c","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516293.053188_134.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.13","timestamp":"2021-03-12T02:31:33.053188210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3Sm3iR_O9gOws1Qm0XHBesI","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.059103_135.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.13","timestamp":"2021-03-12T02:31:33.059103540Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3Sm3iR_O9gOws1Qm0XHBesI","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516293.060699_136.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:33.060699635Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fJHHqFsf0ZnQ8HA3eVv0HsQ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.063543_137.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:33.063543264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fJHHqFsf0ZnQ8HA3eVv0HsQ","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516293.100428_138.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.14","timestamp":"2021-03-12T02:31:33.100428411Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qEcq5zwRQGvPWI89_dceYVx","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.105635_139.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.14","timestamp":"2021-03-12T02:31:33.105635408Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qEcq5zwRQGvPWI89_dceYVx","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516293.107322_140.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:33.107322438Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_usXxiQpihhvHv9w9H2KqHQL","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.112787_141.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:33.112787593Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_usXxiQpihhvHv9w9H2KqHQL","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516293.205051_142.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.15","timestamp":"2021-03-12T02:31:33.205051135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NyLEienNgXaYmXOscru4QK2","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.210392_143.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.15","timestamp":"2021-03-12T02:31:33.210392899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NyLEienNgXaYmXOscru4QK2","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516293.212220_144.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:33.212220343Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_McGu-bUTB7Iwrzro3KX9U6v","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.216631_145.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:33.216631468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_McGu-bUTB7Iwrzro3KX9U6v","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516300.493202_146.1","section":"bucket.instance","name":"bbucnh-7:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.16","timestamp":"2021-03-12T02:31:40.493202575Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJvoXXdfbZSbk6J8UPJzIDB","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.498785_148.1","section":"bucket.instance","name":"bbucnh-7:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.16","timestamp":"2021-03-12T02:31:40.498785950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJvoXXdfbZSbk6J8UPJzIDB","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.500553_149.1","section":"bucket","name":"bbucnh-7","timestamp":"2021-03-12T02:31:40.500553657Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.507000_150.1","section":"bucket","name":"bbucnh-7","timestamp":"2021-03-12T02:31:40.507000938Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.528275_152.1","section":"bucket.instance","name":"bbucnh-8:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.17","timestamp":"2021-03-12T02:31:40.528275074Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n22OShRJQ_FxFcZr65eyDbq","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.533734_153.1","section":"bucket.instance","name":"bbucnh-8:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.17","timestamp":"2021-03-12T02:31:40.533734931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n22OShRJQ_FxFcZr65eyDbq","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.535436_154.1","section":"bucket","name":"bbucnh-8","timestamp":"2021-03-12T02:31:40.535436833Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.541545_155.1","section":"bucket","name":"bbucnh-8","timestamp":"2021-03-12T02:31:40.541545383Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.583042_156.1","section":"bucket.instance","name":"bbucnh-9:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.18","timestamp":"2021-03-12T02:31:40.583042072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WFleci_lBAEwlmYvNA5F_z0","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.587676_157.1","section":"bucket.instance","name":"bbucnh-9:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.18","timestamp":"2021-03-12T02:31:40.587676557Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WFleci_lBAEwlmYvNA5F_z0","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.589578_158.1","section":"bucket","name":"bbucnh-9","timestamp":"2021-03-12T02:31:40.589578574Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.595034_159.1","section":"bucket","name":"bbucnh-9","timestamp":"2021-03-12T02:31:40.595034275Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516309.326188_162.1","section":"bucket","name":"bbucnh-7","timestamp":"2021-03-12T02:31:49.326188604Z","data":{"read_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":1},"write_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":2},"status":{"status":"remove"}}},{"id":"1_1615516309.332337_163.1","section":"bucket","name":"bbucnh-7","timestamp":"2021-03-12T02:31:49.332337164Z","data":{"read_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":1},"write_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516309.508827_165.1","section":"bucket","name":"bbucnh-8","timestamp":"2021-03-12T02:31:49.508827513Z","data":{"read_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":1},"write_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":2},"status":{"status":"remove"}}},{"id":"1_1615516309.514944_166.1","section":"bucket","name":"bbucnh-8","timestamp":"2021-03-12T02:31:49.514944882Z","data":{"read_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":1},"write_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516309.592520_167.1","section":"bucket","name":"bbucnh-9","timestamp":"2021-03-12T02:31:49.592520057Z","data":{"read_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":1},"write_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":2},"status":{"status":"remove"}}},{"id":"1_1615516309.597935_168.1","section":"bucket","name":"bbucnh-9","timestamp":"2021-03-12T02:31:49.597935909Z","data":{"read_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":1},"write_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":2},"status":{"status":"complete"}}}] 2021-03-12T02:40:10.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0cf3ee11-29e6-4776-941b-50bda62adf82', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:10.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0cf3ee11-29e6-4776-941b-50bda62adf82', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:10.210 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0cf3ee11-29e6-4776-941b-50bda62adf82 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:10.254 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:10.254 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:10.462 INFO:teuthology.orchestra.run.smithi114.stdout:[{"id":"1_1615516156.388233_96.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-12T02:29:16.388233974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YL3_4pFMNCff-5Idsb0xw0Y","ver":1},"status":{"status":"write"}}},{"id":"1_1615516156.395861_98.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-12T02:29:16.395861436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YL3_4pFMNCff-5Idsb0xw0Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.517635_58.1","section":"bucket.instance","name":"bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19","timestamp":"2021-03-12T02:31:57.517635735Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jcLKJHBt6Ev7gDksP2Slh3b","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.523069_59.1","section":"bucket.instance","name":"bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19","timestamp":"2021-03-12T02:31:57.523069746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jcLKJHBt6Ev7gDksP2Slh3b","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.525015_60.1","section":"bucket","name":"bbucnh-10","timestamp":"2021-03-12T02:31:57.525015146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6lPftr6jYyKA9E3-g-_Jn6a","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.529693_61.1","section":"bucket","name":"bbucnh-10","timestamp":"2021-03-12T02:31:57.529693741Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6lPftr6jYyKA9E3-g-_Jn6a","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.924299_62.1","section":"bucket.instance","name":"bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20","timestamp":"2021-03-12T02:31:57.924299109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J0Dn8QzeuUr9s9ivyu7dsOD","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.930629_63.1","section":"bucket.instance","name":"bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20","timestamp":"2021-03-12T02:31:57.930629043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J0Dn8QzeuUr9s9ivyu7dsOD","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.932410_64.1","section":"bucket","name":"bbucnh-11","timestamp":"2021-03-12T02:31:57.932410309Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zofeXVzfWAE58gjk_kRUwx9","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.936714_65.1","section":"bucket","name":"bbucnh-11","timestamp":"2021-03-12T02:31:57.936714991Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zofeXVzfWAE58gjk_kRUwx9","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.973905_66.1","section":"bucket.instance","name":"bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21","timestamp":"2021-03-12T02:31:57.973905477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_25ydPCfdlzXA1KPDlkW_pFL","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.978105_67.1","section":"bucket.instance","name":"bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21","timestamp":"2021-03-12T02:31:57.978105914Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_25ydPCfdlzXA1KPDlkW_pFL","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.979768_68.1","section":"bucket","name":"bbucnh-12","timestamp":"2021-03-12T02:31:57.979768274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dw8z_OfXq9g-1vCkmj9nOKx","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.983732_69.1","section":"bucket","name":"bbucnh-12","timestamp":"2021-03-12T02:31:57.983732995Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dw8z_OfXq9g-1vCkmj9nOKx","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516335.993721_86.1","section":"bucket.instance","name":"bbucnh-13:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22","timestamp":"2021-03-12T02:32:15.993721453Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iFsto3hGG7H5MJRUgU2zSGi","ver":1},"status":{"status":"write"}}},{"id":"1_1615516335.999815_87.1","section":"bucket.instance","name":"bbucnh-13:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22","timestamp":"2021-03-12T02:32:15.999815759Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iFsto3hGG7H5MJRUgU2zSGi","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.001721_88.1","section":"bucket","name":"bbucnh-13","timestamp":"2021-03-12T02:32:16.001721436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dy_6thslFFy5DEBDJ7AdBBZ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.006149_89.1","section":"bucket","name":"bbucnh-13","timestamp":"2021-03-12T02:32:16.006149764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dy_6thslFFy5DEBDJ7AdBBZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.023548_90.1","section":"bucket.instance","name":"bbucnh-14:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23","timestamp":"2021-03-12T02:32:16.023548341Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UidaAgtGQcz4suSP3NCWLHf","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.028425_91.1","section":"bucket.instance","name":"bbucnh-14:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23","timestamp":"2021-03-12T02:32:16.028425620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UidaAgtGQcz4suSP3NCWLHf","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.030253_92.1","section":"bucket","name":"bbucnh-14","timestamp":"2021-03-12T02:32:16.030253029Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8u_rE6TyERk7cInsKIFxC_W","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.034867_93.1","section":"bucket","name":"bbucnh-14","timestamp":"2021-03-12T02:32:16.034867278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8u_rE6TyERk7cInsKIFxC_W","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.073203_94.1","section":"bucket.instance","name":"bbucnh-15:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24","timestamp":"2021-03-12T02:32:16.073203702Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fPI6vD6iqij_2lOipNCA5Pj","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.078448_95.1","section":"bucket.instance","name":"bbucnh-15:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24","timestamp":"2021-03-12T02:32:16.078448093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fPI6vD6iqij_2lOipNCA5Pj","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.079985_96.1","section":"bucket","name":"bbucnh-15","timestamp":"2021-03-12T02:32:16.079985141Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZURiB1MDtruSPPRLjEHfFQn","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.084220_97.1","section":"bucket","name":"bbucnh-15","timestamp":"2021-03-12T02:32:16.084220471Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZURiB1MDtruSPPRLjEHfFQn","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.698485_109.1","section":"bucket.instance","name":"bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25","timestamp":"2021-03-12T02:33:02.698485717Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.704669_110.1","section":"bucket.instance","name":"bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25","timestamp":"2021-03-12T02:33:02.704669333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.706515_111.1","section":"bucket","name":"bbucnh-16","timestamp":"2021-03-12T02:33:02.706515387Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3i1dwkEn1BGUdDnt5jjMXCT","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.713003_112.1","section":"bucket","name":"bbucnh-16","timestamp":"2021-03-12T02:33:02.713003405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3i1dwkEn1BGUdDnt5jjMXCT","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.737854_113.1","section":"bucket.instance","name":"bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26","timestamp":"2021-03-12T02:33:02.737854190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.743984_114.1","section":"bucket.instance","name":"bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26","timestamp":"2021-03-12T02:33:02.743984571Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.745895_115.1","section":"bucket","name":"bbucnh-17","timestamp":"2021-03-12T02:33:02.745895289Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cPb986KctBPZNxu1j2usRVk","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.751955_116.1","section":"bucket","name":"bbucnh-17","timestamp":"2021-03-12T02:33:02.751955799Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cPb986KctBPZNxu1j2usRVk","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.793972_117.1","section":"bucket.instance","name":"bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27","timestamp":"2021-03-12T02:33:02.793972919Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.799476_118.1","section":"bucket.instance","name":"bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27","timestamp":"2021-03-12T02:33:02.799476294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.801233_119.1","section":"bucket","name":"bbucnh-18","timestamp":"2021-03-12T02:33:02.801233293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bE0Z6UtgXyyVGQt7bxmNWss","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.805888_120.1","section":"bucket","name":"bbucnh-18","timestamp":"2021-03-12T02:33:02.805888319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bE0Z6UtgXyyVGQt7bxmNWss","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.833909_121.1","section":"bucket.instance","name":"bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25","timestamp":"2021-03-12T02:33:02.833909432Z","data":{"read_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":1},"write_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":2},"status":{"status":"write"}}},{"id":"1_1615516382.839228_122.1","section":"bucket.instance","name":"bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25","timestamp":"2021-03-12T02:33:02.839228126Z","data":{"read_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":1},"write_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516382.844254_123.1","section":"bucket.instance","name":"bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26","timestamp":"2021-03-12T02:33:02.844254956Z","data":{"read_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":1},"write_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":2},"status":{"status":"write"}}},{"id":"1_1615516382.850139_124.1","section":"bucket.instance","name":"bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26","timestamp":"2021-03-12T02:33:02.850139352Z","data":{"read_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":1},"write_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516382.860106_125.1","section":"bucket.instance","name":"bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27","timestamp":"2021-03-12T02:33:02.860106932Z","data":{"read_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":1},"write_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":2},"status":{"status":"write"}}},{"id":"1_1615516382.866387_126.1","section":"bucket.instance","name":"bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27","timestamp":"2021-03-12T02:33:02.866387331Z","data":{"read_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":1},"write_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516477.625386_179.1","section":"bucket.instance","name":"bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28","timestamp":"2021-03-12T02:34:37.625386901Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":1},"status":{"status":"write"}}},{"id":"1_1615516477.630575_180.1","section":"bucket.instance","name":"bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28","timestamp":"2021-03-12T02:34:37.630575238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516477.632196_181.1","section":"bucket","name":"bbucnh-19","timestamp":"2021-03-12T02:34:37.632196980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nPMLF28l579DSvKw8ZKyOBr","ver":1},"status":{"status":"write"}}},{"id":"1_1615516477.636638_182.1","section":"bucket","name":"bbucnh-19","timestamp":"2021-03-12T02:34:37.636638396Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nPMLF28l579DSvKw8ZKyOBr","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516477.641759_183.1","section":"bucket.instance","name":"bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28","timestamp":"2021-03-12T02:34:37.641759734Z","data":{"read_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":1},"write_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":2},"status":{"status":"write"}}},{"id":"1_1615516477.646256_184.1","section":"bucket.instance","name":"bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28","timestamp":"2021-03-12T02:34:37.646256599Z","data":{"read_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":1},"write_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516542.961983_188.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:35:42.961983629Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":1},"status":{"status":"write"}}},{"id":"1_1615516542.966262_189.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:35:42.966262443Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516542.968145_190.1","section":"bucket","name":"bbucnh-20","timestamp":"2021-03-12T02:35:42.968145462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jmYhKySMPNTjtHT_ULsBbuI","ver":1},"status":{"status":"write"}}},{"id":"1_1615516542.972050_191.1","section":"bucket","name":"bbucnh-20","timestamp":"2021-03-12T02:35:42.972050294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jmYhKySMPNTjtHT_ULsBbuI","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516560.741881_192.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:36:00.741881668Z","data":{"read_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":1},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":2},"status":{"status":"write"}}},{"id":"1_1615516560.750253_193.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:36:00.750253702Z","data":{"read_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":1},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516580.510392_194.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:36:20.510392140Z","data":{"read_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":2},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":3},"status":{"status":"write"}}},{"id":"1_1615516580.518628_195.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:36:20.518628079Z","data":{"read_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":2},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":3},"status":{"status":"complete"}}},{"id":"1_1615516601.288975_196.1","section":"bucket.instance","name":"bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30","timestamp":"2021-03-12T02:36:41.288975015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.293430_197.1","section":"bucket.instance","name":"bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30","timestamp":"2021-03-12T02:36:41.293430632Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.295149_198.1","section":"bucket","name":"bbucnh-21","timestamp":"2021-03-12T02:36:41.295149172Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__zd-Z1R4RWVmQc9qnWCZJ9Z","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.301563_199.1","section":"bucket","name":"bbucnh-21","timestamp":"2021-03-12T02:36:41.301563274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__zd-Z1R4RWVmQc9qnWCZJ9Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.326503_200.1","section":"bucket.instance","name":"bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31","timestamp":"2021-03-12T02:36:41.326503965Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.331867_201.1","section":"bucket.instance","name":"bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31","timestamp":"2021-03-12T02:36:41.331867729Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.333775_202.1","section":"bucket","name":"bbucnh-22","timestamp":"2021-03-12T02:36:41.333775173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lRVZbkb3M5VDTFzWzb2Ao_n","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.338365_203.1","section":"bucket","name":"bbucnh-22","timestamp":"2021-03-12T02:36:41.338365599Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lRVZbkb3M5VDTFzWzb2Ao_n","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.375262_204.1","section":"bucket.instance","name":"bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32","timestamp":"2021-03-12T02:36:41.375262696Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.380184_205.1","section":"bucket.instance","name":"bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32","timestamp":"2021-03-12T02:36:41.380184001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.382095_206.1","section":"bucket","name":"bbucnh-23","timestamp":"2021-03-12T02:36:41.382095207Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ox0KlvX6xNF4jdqzrljR_Yz","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.386448_207.1","section":"bucket","name":"bbucnh-23","timestamp":"2021-03-12T02:36:41.386448784Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ox0KlvX6xNF4jdqzrljR_Yz","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.406400_208.1","section":"bucket.instance","name":"bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30","timestamp":"2021-03-12T02:36:41.406400290Z","data":{"read_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":1},"write_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":2},"status":{"status":"write"}}},{"id":"1_1615516601.411130_209.1","section":"bucket.instance","name":"bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30","timestamp":"2021-03-12T02:36:41.411130339Z","data":{"read_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":1},"write_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516601.415296_210.1","section":"bucket.instance","name":"bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31","timestamp":"2021-03-12T02:36:41.415296943Z","data":{"read_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":1},"write_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":2},"status":{"status":"write"}}},{"id":"1_1615516601.420232_211.1","section":"bucket.instance","name":"bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31","timestamp":"2021-03-12T02:36:41.420232785Z","data":{"read_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":1},"write_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516601.429424_212.1","section":"bucket.instance","name":"bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32","timestamp":"2021-03-12T02:36:41.429424011Z","data":{"read_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":1},"write_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":2},"status":{"status":"write"}}},{"id":"1_1615516601.435061_213.1","section":"bucket.instance","name":"bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32","timestamp":"2021-03-12T02:36:41.435061595Z","data":{"read_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":1},"write_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516623.905134_226.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:03.905134670Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":1},"status":{"status":"write"}}},{"id":"1_1615516623.909582_227.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:03.909582485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516623.911054_228.1","section":"bucket","name":"bbucnh-24","timestamp":"2021-03-12T02:37:03.911054290Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kAh_Dud7mDE7666DS44bDVH","ver":1},"status":{"status":"write"}}},{"id":"1_1615516623.915168_229.1","section":"bucket","name":"bbucnh-24","timestamp":"2021-03-12T02:37:03.915168538Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kAh_Dud7mDE7666DS44bDVH","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516623.938779_230.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:03.938779082Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":1},"status":{"status":"write"}}},{"id":"1_1615516623.944057_231.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:03.944057002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516623.945869_232.1","section":"bucket","name":"bbucnh-25","timestamp":"2021-03-12T02:37:03.945869854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FVWFGEedKW_N26l2XXHS__P","ver":1},"status":{"status":"write"}}},{"id":"1_1615516623.952222_233.1","section":"bucket","name":"bbucnh-25","timestamp":"2021-03-12T02:37:03.952222510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FVWFGEedKW_N26l2XXHS__P","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516623.998369_234.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:03.998369746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":1},"status":{"status":"write"}}},{"id":"1_1615516624.003251_235.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.003251527Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516624.004804_236.1","section":"bucket","name":"bbucnh-26","timestamp":"2021-03-12T02:37:04.004804195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pZWbNId4gDq01c4QPLoSkT3","ver":1},"status":{"status":"write"}}},{"id":"1_1615516624.009867_237.1","section":"bucket","name":"bbucnh-26","timestamp":"2021-03-12T02:37:04.009867850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pZWbNId4gDq01c4QPLoSkT3","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516624.034129_238.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:04.034129873Z","data":{"read_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":1},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":2},"status":{"status":"write"}}},{"id":"1_1615516624.040808_239.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:04.040808023Z","data":{"read_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":1},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516624.044134_240.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:04.044134611Z","data":{"read_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":2},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":3},"status":{"status":"write"}}},{"id":"1_1615516624.051708_241.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:04.051708030Z","data":{"read_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":2},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":3},"status":{"status":"complete"}}},{"id":"1_1615516624.056887_242.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:04.056887996Z","data":{"read_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":1},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":2},"status":{"status":"write"}}},{"id":"1_1615516624.063220_243.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:04.063220593Z","data":{"read_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":1},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516624.073512_244.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:04.073512038Z","data":{"read_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":2},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":3},"status":{"status":"write"}}},{"id":"1_1615516624.080300_245.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:04.080300997Z","data":{"read_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":2},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":3},"status":{"status":"complete"}}},{"id":"1_1615516624.092046_246.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.092046934Z","data":{"read_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":1},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":2},"status":{"status":"write"}}},{"id":"1_1615516624.098396_247.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.098396048Z","data":{"read_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":1},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516624.109918_248.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.109918336Z","data":{"read_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":2},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":3},"status":{"status":"write"}}},{"id":"1_1615516624.119886_249.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.119886963Z","data":{"read_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":2},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":3},"status":{"status":"complete"}}},{"id":"1_1615516648.953955_260.1","section":"bucket.instance","name":"bbucnh-27:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.36","timestamp":"2021-03-12T02:37:28.953955696Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516648.958378_261.1","section":"bucket.instance","name":"bbucnh-27:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.36","timestamp":"2021-03-12T02:37:28.958378293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516648.959965_262.1","section":"bucket","name":"bbucnh-27","timestamp":"2021-03-12T02:37:28.959965688Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FmfQYONhZDXgO1uw7EYs9_S","ver":1},"status":{"status":"write"}}},{"id":"1_1615516648.963902_263.1","section":"bucket","name":"bbucnh-27","timestamp":"2021-03-12T02:37:28.963902711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FmfQYONhZDXgO1uw7EYs9_S","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516648.987286_264.1","section":"bucket.instance","name":"bbucnh-28:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37","timestamp":"2021-03-12T02:37:28.987286166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":1},"status":{"status":"write"}}},{"id":"1_1615516648.993373_265.1","section":"bucket.instance","name":"bbucnh-28:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37","timestamp":"2021-03-12T02:37:28.993373542Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516648.995310_266.1","section":"bucket","name":"bbucnh-28","timestamp":"2021-03-12T02:37:28.995310434Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2gVXe3dHFNHJj_5mBzi7BFQ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.000050_267.1","section":"bucket","name":"bbucnh-28","timestamp":"2021-03-12T02:37:29.000050799Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2gVXe3dHFNHJj_5mBzi7BFQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.040216_268.1","section":"bucket.instance","name":"bbucnh-29:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38","timestamp":"2021-03-12T02:37:29.040216016Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.045685_269.1","section":"bucket.instance","name":"bbucnh-29:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38","timestamp":"2021-03-12T02:37:29.045685128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.047750_270.1","section":"bucket","name":"bbucnh-29","timestamp":"2021-03-12T02:37:29.047750134Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NzPM--eEDeYlpvhbh0MdeK4","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.052200_271.1","section":"bucket","name":"bbucnh-29","timestamp":"2021-03-12T02:37:29.052200209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NzPM--eEDeYlpvhbh0MdeK4","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.075303_272.1","section":"bucket.instance","name":"bbucnh-27:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.36","timestamp":"2021-03-12T02:37:29.075303587Z","data":{"read_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":1},"write_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.080878_273.1","section":"bucket.instance","name":"bbucnh-27:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.36","timestamp":"2021-03-12T02:37:29.080878607Z","data":{"read_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":1},"write_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.129020_274.1","section":"bucket.instance","name":"bbucnh-28:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37","timestamp":"2021-03-12T02:37:29.129020143Z","data":{"read_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":1},"write_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.137207_275.1","section":"bucket.instance","name":"bbucnh-28:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37","timestamp":"2021-03-12T02:37:29.137207384Z","data":{"read_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":1},"write_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.196723_276.1","section":"bucket.instance","name":"bbucnh-29:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38","timestamp":"2021-03-12T02:37:29.196723737Z","data":{"read_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":1},"write_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.204941_277.1","section":"bucket.instance","name":"bbucnh-29:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38","timestamp":"2021-03-12T02:37:29.204941290Z","data":{"read_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":1},"write_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.273026_278.1","section":"bucket.instance","name":"bbucnh-30:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.39","timestamp":"2021-03-12T02:37:29.273026550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.278145_279.1","section":"bucket.instance","name":"bbucnh-30:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.39","timestamp":"2021-03-12T02:37:29.278145289Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.279975_280.1","section":"bucket","name":"bbucnh-30","timestamp":"2021-03-12T02:37:29.279975374Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ktkorz1HBIgiOiSGUck37Fi","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.284871_281.1","section":"bucket","name":"bbucnh-30","timestamp":"2021-03-12T02:37:29.284871749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ktkorz1HBIgiOiSGUck37Fi","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.312142_282.1","section":"bucket.instance","name":"bbucnh-31:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40","timestamp":"2021-03-12T02:37:29.312142460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.317109_283.1","section":"bucket.instance","name":"bbucnh-31:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40","timestamp":"2021-03-12T02:37:29.317109530Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.318899_284.1","section":"bucket","name":"bbucnh-31","timestamp":"2021-03-12T02:37:29.318899628Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RVIUeIv4NvsRyQ3Z9VfRls-","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.323537_285.1","section":"bucket","name":"bbucnh-31","timestamp":"2021-03-12T02:37:29.323537422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RVIUeIv4NvsRyQ3Z9VfRls-","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.364204_286.1","section":"bucket.instance","name":"bbucnh-32:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41","timestamp":"2021-03-12T02:37:29.364204861Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.369006_287.1","section":"bucket.instance","name":"bbucnh-32:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41","timestamp":"2021-03-12T02:37:29.369006270Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.370821_288.1","section":"bucket","name":"bbucnh-32","timestamp":"2021-03-12T02:37:29.370821226Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uNNdkr5c4kYGRpyB0KL4L6X","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.374864_289.1","section":"bucket","name":"bbucnh-32","timestamp":"2021-03-12T02:37:29.374864824Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uNNdkr5c4kYGRpyB0KL4L6X","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.400473_291.1","section":"bucket.instance","name":"bbucnh-30:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.39","timestamp":"2021-03-12T02:37:29.400473777Z","data":{"read_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":1},"write_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.405529_292.1","section":"bucket.instance","name":"bbucnh-30:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.39","timestamp":"2021-03-12T02:37:29.405529038Z","data":{"read_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":1},"write_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.454396_293.1","section":"bucket.instance","name":"bbucnh-31:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40","timestamp":"2021-03-12T02:37:29.454396735Z","data":{"read_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":1},"write_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.462159_294.1","section":"bucket.instance","name":"bbucnh-31:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40","timestamp":"2021-03-12T02:37:29.462159544Z","data":{"read_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":1},"write_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.526776_295.1","section":"bucket.instance","name":"bbucnh-32:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41","timestamp":"2021-03-12T02:37:29.526776297Z","data":{"read_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":1},"write_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.534441_296.1","section":"bucket.instance","name":"bbucnh-32:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41","timestamp":"2021-03-12T02:37:29.534441885Z","data":{"read_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":1},"write_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.609051_298.1","section":"bucket.instance","name":"bbucnh-33:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.42","timestamp":"2021-03-12T02:37:29.609051947Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.613737_299.1","section":"bucket.instance","name":"bbucnh-33:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.42","timestamp":"2021-03-12T02:37:29.613737442Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.615385_300.1","section":"bucket","name":"bbucnh-33","timestamp":"2021-03-12T02:37:29.615385271Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_15Kd9p4ie5l3fyoCESSGjmB","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.619356_301.1","section":"bucket","name":"bbucnh-33","timestamp":"2021-03-12T02:37:29.619356209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_15Kd9p4ie5l3fyoCESSGjmB","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.644293_302.1","section":"bucket.instance","name":"bbucnh-34:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43","timestamp":"2021-03-12T02:37:29.644293956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.649652_303.1","section":"bucket.instance","name":"bbucnh-34:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43","timestamp":"2021-03-12T02:37:29.649652088Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.651633_304.1","section":"bucket","name":"bbucnh-34","timestamp":"2021-03-12T02:37:29.651633615Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KxY1ghborztF7ijmoRBmfYR","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.656854_305.1","section":"bucket","name":"bbucnh-34","timestamp":"2021-03-12T02:37:29.656854451Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KxY1ghborztF7ijmoRBmfYR","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.700347_306.1","section":"bucket.instance","name":"bbucnh-35:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44","timestamp":"2021-03-12T02:37:29.700347440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.705720_307.1","section":"bucket.instance","name":"bbucnh-35:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44","timestamp":"2021-03-12T02:37:29.705720561Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.707370_308.1","section":"bucket","name":"bbucnh-35","timestamp":"2021-03-12T02:37:29.707370179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tJebU3psH_SbzeVFZfWinUh","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.711408_309.1","section":"bucket","name":"bbucnh-35","timestamp":"2021-03-12T02:37:29.711408051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tJebU3psH_SbzeVFZfWinUh","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.734891_310.1","section":"bucket.instance","name":"bbucnh-33:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.42","timestamp":"2021-03-12T02:37:29.734891456Z","data":{"read_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":1},"write_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.739920_311.1","section":"bucket.instance","name":"bbucnh-33:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.42","timestamp":"2021-03-12T02:37:29.739920185Z","data":{"read_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":1},"write_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516650.789737_312.1","section":"bucket.instance","name":"bbucnh-34:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43","timestamp":"2021-03-12T02:37:30.789737190Z","data":{"read_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":1},"write_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":2},"status":{"status":"write"}}},{"id":"1_1615516650.798206_313.1","section":"bucket.instance","name":"bbucnh-34:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43","timestamp":"2021-03-12T02:37:30.798206828Z","data":{"read_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":1},"write_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516652.335363_314.1","section":"bucket.instance","name":"bbucnh-35:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44","timestamp":"2021-03-12T02:37:32.335363586Z","data":{"read_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":1},"write_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":2},"status":{"status":"write"}}},{"id":"1_1615516652.344811_315.1","section":"bucket.instance","name":"bbucnh-35:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44","timestamp":"2021-03-12T02:37:32.344811119Z","data":{"read_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":1},"write_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516653.069940_316.1","section":"bucket.instance","name":"bbucnh-36:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45","timestamp":"2021-03-12T02:37:33.069940667Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VuDCjriGUmI_jGsCkSwyMPM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516653.075083_317.1","section":"bucket.instance","name":"bbucnh-36:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45","timestamp":"2021-03-12T02:37:33.075083097Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VuDCjriGUmI_jGsCkSwyMPM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516653.076710_318.1","section":"bucket","name":"bbucnh-36","timestamp":"2021-03-12T02:37:33.076710866Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6lGSmQvlrfayDP125PAmHZF","ver":1},"status":{"status":"write"}}},{"id":"1_1615516653.081538_319.1","section":"bucket","name":"bbucnh-36","timestamp":"2021-03-12T02:37:33.081538100Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6lGSmQvlrfayDP125PAmHZF","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516653.664443_320.1","section":"bucket.instance","name":"bbucnh-37:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46","timestamp":"2021-03-12T02:37:33.664443824Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HmfgDDWFN0bOGRAMO2ByNk7","ver":1},"status":{"status":"write"}}},{"id":"1_1615516653.669629_321.1","section":"bucket.instance","name":"bbucnh-37:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46","timestamp":"2021-03-12T02:37:33.669629721Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HmfgDDWFN0bOGRAMO2ByNk7","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516653.671304_322.1","section":"bucket","name":"bbucnh-37","timestamp":"2021-03-12T02:37:33.671304239Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J6fc8NV-QkiOt2H3vUtcdSS","ver":1},"status":{"status":"write"}}},{"id":"1_1615516653.675804_323.1","section":"bucket","name":"bbucnh-37","timestamp":"2021-03-12T02:37:33.675804856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J6fc8NV-QkiOt2H3vUtcdSS","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516653.998533_324.1","section":"bucket.instance","name":"bbucnh-38:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47","timestamp":"2021-03-12T02:37:33.998533211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DXFrPBq7cR1YlfkiP9rTC_h","ver":1},"status":{"status":"write"}}},{"id":"1_1615516654.002986_325.1","section":"bucket.instance","name":"bbucnh-38:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47","timestamp":"2021-03-12T02:37:34.002986261Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DXFrPBq7cR1YlfkiP9rTC_h","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516654.004526_326.1","section":"bucket","name":"bbucnh-38","timestamp":"2021-03-12T02:37:34.004526175Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B2U5VzBNff39d-uvhTwPTVA","ver":1},"status":{"status":"write"}}},{"id":"1_1615516654.008644_327.1","section":"bucket","name":"bbucnh-38","timestamp":"2021-03-12T02:37:34.008644344Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B2U5VzBNff39d-uvhTwPTVA","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.472251_328.1","section":"bucket.instance","name":"bbucnh-39:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.48","timestamp":"2021-03-12T02:37:36.472251903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxwXPsBw5MBOuLy2DL0RzSC","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.477307_329.1","section":"bucket.instance","name":"bbucnh-39:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.48","timestamp":"2021-03-12T02:37:36.477307238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxwXPsBw5MBOuLy2DL0RzSC","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.479321_330.1","section":"bucket","name":"bbucnh-39","timestamp":"2021-03-12T02:37:36.479321851Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7RmOoO6yS0MAc_ELYlYq8yN","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.483505_331.1","section":"bucket","name":"bbucnh-39","timestamp":"2021-03-12T02:37:36.483505123Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7RmOoO6yS0MAc_ELYlYq8yN","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.511042_332.1","section":"bucket.instance","name":"bbucnh-40:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.49","timestamp":"2021-03-12T02:37:36.511042128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q1cAgCTgGfyj4SRitg3YHYj","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.516504_333.1","section":"bucket.instance","name":"bbucnh-40:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.49","timestamp":"2021-03-12T02:37:36.516504986Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q1cAgCTgGfyj4SRitg3YHYj","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.518471_334.1","section":"bucket","name":"bbucnh-40","timestamp":"2021-03-12T02:37:36.518471862Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vmcO4AQPJh3F7zPA8Q1MDcO","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.522955_335.1","section":"bucket","name":"bbucnh-40","timestamp":"2021-03-12T02:37:36.522955683Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vmcO4AQPJh3F7zPA8Q1MDcO","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.569936_336.1","section":"bucket.instance","name":"bbucnh-41:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.50","timestamp":"2021-03-12T02:37:36.569936921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V9oej-UfPhBNN29T2LbubWl","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.574506_337.1","section":"bucket.instance","name":"bbucnh-41:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.50","timestamp":"2021-03-12T02:37:36.574506107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V9oej-UfPhBNN29T2LbubWl","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.576240_338.1","section":"bucket","name":"bbucnh-41","timestamp":"2021-03-12T02:37:36.576240547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TmSKoQq5CYyZMuTTOWMF8jB","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.580426_339.1","section":"bucket","name":"bbucnh-41","timestamp":"2021-03-12T02:37:36.580426570Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TmSKoQq5CYyZMuTTOWMF8jB","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516277.246267_81.1","section":"bucket.instance","name":"bbucnh-1:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.1","timestamp":"2021-03-12T02:31:17.246267120Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qfxaS5zdrJis58Zg6Obg75g","ver":1},"status":{"status":"write"}}},{"id":"1_1615516277.255262_85.1","section":"bucket.instance","name":"bbucnh-1:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.1","timestamp":"2021-03-12T02:31:17.255262459Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qfxaS5zdrJis58Zg6Obg75g","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516277.256664_86.1","section":"bucket","name":"bbucnh-1","timestamp":"2021-03-12T02:31:17.256664887Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SJQ8Ox0nk_KuM-tokbc3CHJ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516277.260851_87.1","section":"bucket","name":"bbucnh-1","timestamp":"2021-03-12T02:31:17.260851736Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SJQ8Ox0nk_KuM-tokbc3CHJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516277.279301_88.1","section":"bucket.instance","name":"bbucnh-2:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.2","timestamp":"2021-03-12T02:31:17.279301358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OdmYkD6xh5KguUnb-mdZ1aM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516277.284169_89.1","section":"bucket.instance","name":"bbucnh-2:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.2","timestamp":"2021-03-12T02:31:17.284169621Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OdmYkD6xh5KguUnb-mdZ1aM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516277.285818_90.1","section":"bucket","name":"bbucnh-2","timestamp":"2021-03-12T02:31:17.285818007Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U61zBQN8Y4-IaOqXqUfnFDK","ver":1},"status":{"status":"write"}}},{"id":"1_1615516277.289902_91.1","section":"bucket","name":"bbucnh-2","timestamp":"2021-03-12T02:31:17.289902101Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U61zBQN8Y4-IaOqXqUfnFDK","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516280.582981_92.1","section":"bucket.instance","name":"bbucnh-3:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.3","timestamp":"2021-03-12T02:31:20.582981563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WmExOXe_DXwUuw8ZWwStmCG","ver":1},"status":{"status":"write"}}},{"id":"1_1615516280.587282_93.1","section":"bucket.instance","name":"bbucnh-3:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.3","timestamp":"2021-03-12T02:31:20.587282731Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WmExOXe_DXwUuw8ZWwStmCG","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516280.588897_94.1","section":"bucket","name":"bbucnh-3","timestamp":"2021-03-12T02:31:20.588897050Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vTlspttjgtqvLuUNPCk_X1Z","ver":1},"status":{"status":"write"}}},{"id":"1_1615516280.593132_95.1","section":"bucket","name":"bbucnh-3","timestamp":"2021-03-12T02:31:20.593132328Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vTlspttjgtqvLuUNPCk_X1Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516289.394488_96.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.4","timestamp":"2021-03-12T02:31:29.394488830Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P9XE_bWiNIw3vvd5uTEfxr3","ver":1},"status":{"status":"write"}}},{"id":"1_1615516289.399511_98.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.4","timestamp":"2021-03-12T02:31:29.399511189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P9XE_bWiNIw3vvd5uTEfxr3","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516289.400904_99.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:29.400904315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HJA5lWRMWyY0IaqyBfFi1N4","ver":1},"status":{"status":"write"}}},{"id":"1_1615516289.404420_100.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:29.404420265Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HJA5lWRMWyY0IaqyBfFi1N4","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516289.903936_101.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.5","timestamp":"2021-03-12T02:31:29.903936280Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1ecwCnmWKxBMRrpYQmvXMe-","ver":1},"status":{"status":"write"}}},{"id":"1_1615516289.908750_102.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.5","timestamp":"2021-03-12T02:31:29.908750593Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1ecwCnmWKxBMRrpYQmvXMe-","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516289.910808_103.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:29.910808743Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nZiOpvQgZemMdcUiOXIn1VT","ver":1},"status":{"status":"write"}}},{"id":"1_1615516289.916752_104.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:29.916752691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nZiOpvQgZemMdcUiOXIn1VT","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516290.449434_105.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.6","timestamp":"2021-03-12T02:31:30.449434628Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rHxZdUR046Wgm3Zafw0Xi8l","ver":1},"status":{"status":"write"}}},{"id":"1_1615516290.453654_106.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.6","timestamp":"2021-03-12T02:31:30.453654797Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rHxZdUR046Wgm3Zafw0Xi8l","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516290.455139_107.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:30.455139379Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sg78H-Zgxy2C6QwruYgc-Rg","ver":1},"status":{"status":"write"}}},{"id":"1_1615516290.459112_108.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:30.459112128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sg78H-Zgxy2C6QwruYgc-Rg","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.676123_109.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.7","timestamp":"2021-03-12T02:31:32.676123449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T4BoTd-m-PH42nJonMN7kZP","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.681307_111.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.7","timestamp":"2021-03-12T02:31:32.681307323Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T4BoTd-m-PH42nJonMN7kZP","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.683099_112.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:32.683099550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EBuzwkBweLN5rvCeTKt4REO","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.688466_113.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:32.688466865Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EBuzwkBweLN5rvCeTKt4REO","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.712314_114.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.8","timestamp":"2021-03-12T02:31:32.712314715Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c8czZW2QuWkq0wJE4l-Nvwz","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.716801_115.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.8","timestamp":"2021-03-12T02:31:32.716801906Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c8czZW2QuWkq0wJE4l-Nvwz","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.718398_116.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:32.718398180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yjAubSLjTLPIuEzKEWdFJD8","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.721131_117.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:32.721131522Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yjAubSLjTLPIuEzKEWdFJD8","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.742288_118.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.9","timestamp":"2021-03-12T02:31:32.742288024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cd_ZmHPW-yz7g9wMfu8z1OM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.746471_119.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.9","timestamp":"2021-03-12T02:31:32.746471019Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cd_ZmHPW-yz7g9wMfu8z1OM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.748067_120.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:32.748067528Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yyCMzqqMnUhRq-wn-e0Oe_1","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.752347_121.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:32.752347222Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yyCMzqqMnUhRq-wn-e0Oe_1","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.782913_122.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.10","timestamp":"2021-03-12T02:31:32.782913622Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cbSranN-_o_cTrR-FcWF_2Y","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.788130_123.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.10","timestamp":"2021-03-12T02:31:32.788130810Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cbSranN-_o_cTrR-FcWF_2Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.790158_124.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:32.790158153Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7PSv7RO8pR2twgmTFBWqx0N","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.795541_125.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:32.795541882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7PSv7RO8pR2twgmTFBWqx0N","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.844603_126.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.11","timestamp":"2021-03-12T02:31:32.844603751Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4_B8b59PwgOUx0apcPz4YE3","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.849848_127.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.11","timestamp":"2021-03-12T02:31:32.849848924Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4_B8b59PwgOUx0apcPz4YE3","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.851664_128.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:32.851664510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G_5ocbuuJ8e81M9Oey3KkUa","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.855275_129.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:32.855275037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G_5ocbuuJ8e81M9Oey3KkUa","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.947746_130.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.12","timestamp":"2021-03-12T02:31:32.947746750Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hLAW_-1vXKVBgOh8Z3yp4RR","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.952705_131.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.12","timestamp":"2021-03-12T02:31:32.952705876Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hLAW_-1vXKVBgOh8Z3yp4RR","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.954374_132.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:32.954374195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u-RVInQ17b5MGi2bJAMXC9c","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.957220_133.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:32.957220699Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u-RVInQ17b5MGi2bJAMXC9c","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516293.053188_134.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.13","timestamp":"2021-03-12T02:31:33.053188210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3Sm3iR_O9gOws1Qm0XHBesI","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.059103_135.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.13","timestamp":"2021-03-12T02:31:33.059103540Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3Sm3iR_O9gOws1Qm0XHBesI","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516293.060699_136.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:33.060699635Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fJHHqFsf0ZnQ8HA3eVv0HsQ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.063543_137.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:33.063543264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fJHHqFsf0ZnQ8HA3eVv0HsQ","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516293.100428_138.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.14","timestamp":"2021-03-12T02:31:33.100428411Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qEcq5zwRQGvPWI89_dceYVx","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.105635_139.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.14","timestamp":"2021-03-12T02:31:33.105635408Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qEcq5zwRQGvPWI89_dceYVx","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516293.107322_140.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:33.107322438Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_usXxiQpihhvHv9w9H2KqHQL","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.112787_141.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:33.112787593Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_usXxiQpihhvHv9w9H2KqHQL","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516293.205051_142.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.15","timestamp":"2021-03-12T02:31:33.205051135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NyLEienNgXaYmXOscru4QK2","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.210392_143.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.15","timestamp":"2021-03-12T02:31:33.210392899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NyLEienNgXaYmXOscru4QK2","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516293.212220_144.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:33.212220343Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_McGu-bUTB7Iwrzro3KX9U6v","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.216631_145.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:33.216631468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_McGu-bUTB7Iwrzro3KX9U6v","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516300.493202_146.1","section":"bucket.instance","name":"bbucnh-7:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.16","timestamp":"2021-03-12T02:31:40.493202575Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJvoXXdfbZSbk6J8UPJzIDB","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.498785_148.1","section":"bucket.instance","name":"bbucnh-7:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.16","timestamp":"2021-03-12T02:31:40.498785950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJvoXXdfbZSbk6J8UPJzIDB","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.500553_149.1","section":"bucket","name":"bbucnh-7","timestamp":"2021-03-12T02:31:40.500553657Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.507000_150.1","section":"bucket","name":"bbucnh-7","timestamp":"2021-03-12T02:31:40.507000938Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.528275_152.1","section":"bucket.instance","name":"bbucnh-8:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.17","timestamp":"2021-03-12T02:31:40.528275074Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n22OShRJQ_FxFcZr65eyDbq","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.533734_153.1","section":"bucket.instance","name":"bbucnh-8:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.17","timestamp":"2021-03-12T02:31:40.533734931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n22OShRJQ_FxFcZr65eyDbq","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.535436_154.1","section":"bucket","name":"bbucnh-8","timestamp":"2021-03-12T02:31:40.535436833Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.541545_155.1","section":"bucket","name":"bbucnh-8","timestamp":"2021-03-12T02:31:40.541545383Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.583042_156.1","section":"bucket.instance","name":"bbucnh-9:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.18","timestamp":"2021-03-12T02:31:40.583042072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WFleci_lBAEwlmYvNA5F_z0","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.587676_157.1","section":"bucket.instance","name":"bbucnh-9:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.18","timestamp":"2021-03-12T02:31:40.587676557Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WFleci_lBAEwlmYvNA5F_z0","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.589578_158.1","section":"bucket","name":"bbucnh-9","timestamp":"2021-03-12T02:31:40.589578574Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.595034_159.1","section":"bucket","name":"bbucnh-9","timestamp":"2021-03-12T02:31:40.595034275Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516309.326188_162.1","section":"bucket","name":"bbucnh-7","timestamp":"2021-03-12T02:31:49.326188604Z","data":{"read_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":1},"write_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":2},"status":{"status":"remove"}}},{"id":"1_1615516309.332337_163.1","section":"bucket","name":"bbucnh-7","timestamp":"2021-03-12T02:31:49.332337164Z","data":{"read_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":1},"write_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516309.508827_165.1","section":"bucket","name":"bbucnh-8","timestamp":"2021-03-12T02:31:49.508827513Z","data":{"read_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":1},"write_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":2},"status":{"status":"remove"}}},{"id":"1_1615516309.514944_166.1","section":"bucket","name":"bbucnh-8","timestamp":"2021-03-12T02:31:49.514944882Z","data":{"read_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":1},"write_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516309.592520_167.1","section":"bucket","name":"bbucnh-9","timestamp":"2021-03-12T02:31:49.592520057Z","data":{"read_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":1},"write_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":2},"status":{"status":"remove"}}},{"id":"1_1615516309.597935_168.1","section":"bucket","name":"bbucnh-9","timestamp":"2021-03-12T02:31:49.597935909Z","data":{"read_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":1},"write_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":2},"status":{"status":"complete"}}}] 2021-03-12T02:40:10.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0cf3ee11-29e6-4776-941b-50bda62adf82', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:10.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0cf3ee11-29e6-4776-941b-50bda62adf82', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:10.465 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0cf3ee11-29e6-4776-941b-50bda62adf82 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:10.503 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:10.503 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:10.724 INFO:teuthology.orchestra.run.smithi077.stdout:[{"id":"1_1615516156.388233_96.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-12T02:29:16.388233974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YL3_4pFMNCff-5Idsb0xw0Y","ver":1},"status":{"status":"write"}}},{"id":"1_1615516156.395861_98.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-12T02:29:16.395861436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YL3_4pFMNCff-5Idsb0xw0Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.517635_58.1","section":"bucket.instance","name":"bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19","timestamp":"2021-03-12T02:31:57.517635735Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jcLKJHBt6Ev7gDksP2Slh3b","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.523069_59.1","section":"bucket.instance","name":"bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19","timestamp":"2021-03-12T02:31:57.523069746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jcLKJHBt6Ev7gDksP2Slh3b","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.525015_60.1","section":"bucket","name":"bbucnh-10","timestamp":"2021-03-12T02:31:57.525015146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6lPftr6jYyKA9E3-g-_Jn6a","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.529693_61.1","section":"bucket","name":"bbucnh-10","timestamp":"2021-03-12T02:31:57.529693741Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6lPftr6jYyKA9E3-g-_Jn6a","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.924299_62.1","section":"bucket.instance","name":"bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20","timestamp":"2021-03-12T02:31:57.924299109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J0Dn8QzeuUr9s9ivyu7dsOD","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.930629_63.1","section":"bucket.instance","name":"bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20","timestamp":"2021-03-12T02:31:57.930629043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J0Dn8QzeuUr9s9ivyu7dsOD","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.932410_64.1","section":"bucket","name":"bbucnh-11","timestamp":"2021-03-12T02:31:57.932410309Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zofeXVzfWAE58gjk_kRUwx9","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.936714_65.1","section":"bucket","name":"bbucnh-11","timestamp":"2021-03-12T02:31:57.936714991Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zofeXVzfWAE58gjk_kRUwx9","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.973905_66.1","section":"bucket.instance","name":"bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21","timestamp":"2021-03-12T02:31:57.973905477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_25ydPCfdlzXA1KPDlkW_pFL","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.978105_67.1","section":"bucket.instance","name":"bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21","timestamp":"2021-03-12T02:31:57.978105914Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_25ydPCfdlzXA1KPDlkW_pFL","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.979768_68.1","section":"bucket","name":"bbucnh-12","timestamp":"2021-03-12T02:31:57.979768274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dw8z_OfXq9g-1vCkmj9nOKx","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.983732_69.1","section":"bucket","name":"bbucnh-12","timestamp":"2021-03-12T02:31:57.983732995Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dw8z_OfXq9g-1vCkmj9nOKx","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516335.993721_86.1","section":"bucket.instance","name":"bbucnh-13:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22","timestamp":"2021-03-12T02:32:15.993721453Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iFsto3hGG7H5MJRUgU2zSGi","ver":1},"status":{"status":"write"}}},{"id":"1_1615516335.999815_87.1","section":"bucket.instance","name":"bbucnh-13:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22","timestamp":"2021-03-12T02:32:15.999815759Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iFsto3hGG7H5MJRUgU2zSGi","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.001721_88.1","section":"bucket","name":"bbucnh-13","timestamp":"2021-03-12T02:32:16.001721436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dy_6thslFFy5DEBDJ7AdBBZ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.006149_89.1","section":"bucket","name":"bbucnh-13","timestamp":"2021-03-12T02:32:16.006149764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dy_6thslFFy5DEBDJ7AdBBZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.023548_90.1","section":"bucket.instance","name":"bbucnh-14:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23","timestamp":"2021-03-12T02:32:16.023548341Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UidaAgtGQcz4suSP3NCWLHf","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.028425_91.1","section":"bucket.instance","name":"bbucnh-14:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23","timestamp":"2021-03-12T02:32:16.028425620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UidaAgtGQcz4suSP3NCWLHf","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.030253_92.1","section":"bucket","name":"bbucnh-14","timestamp":"2021-03-12T02:32:16.030253029Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8u_rE6TyERk7cInsKIFxC_W","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.034867_93.1","section":"bucket","name":"bbucnh-14","timestamp":"2021-03-12T02:32:16.034867278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8u_rE6TyERk7cInsKIFxC_W","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.073203_94.1","section":"bucket.instance","name":"bbucnh-15:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24","timestamp":"2021-03-12T02:32:16.073203702Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fPI6vD6iqij_2lOipNCA5Pj","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.078448_95.1","section":"bucket.instance","name":"bbucnh-15:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24","timestamp":"2021-03-12T02:32:16.078448093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fPI6vD6iqij_2lOipNCA5Pj","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.079985_96.1","section":"bucket","name":"bbucnh-15","timestamp":"2021-03-12T02:32:16.079985141Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZURiB1MDtruSPPRLjEHfFQn","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.084220_97.1","section":"bucket","name":"bbucnh-15","timestamp":"2021-03-12T02:32:16.084220471Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZURiB1MDtruSPPRLjEHfFQn","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.698485_109.1","section":"bucket.instance","name":"bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25","timestamp":"2021-03-12T02:33:02.698485717Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.704669_110.1","section":"bucket.instance","name":"bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25","timestamp":"2021-03-12T02:33:02.704669333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.706515_111.1","section":"bucket","name":"bbucnh-16","timestamp":"2021-03-12T02:33:02.706515387Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3i1dwkEn1BGUdDnt5jjMXCT","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.713003_112.1","section":"bucket","name":"bbucnh-16","timestamp":"2021-03-12T02:33:02.713003405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3i1dwkEn1BGUdDnt5jjMXCT","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.737854_113.1","section":"bucket.instance","name":"bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26","timestamp":"2021-03-12T02:33:02.737854190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.743984_114.1","section":"bucket.instance","name":"bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26","timestamp":"2021-03-12T02:33:02.743984571Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.745895_115.1","section":"bucket","name":"bbucnh-17","timestamp":"2021-03-12T02:33:02.745895289Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cPb986KctBPZNxu1j2usRVk","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.751955_116.1","section":"bucket","name":"bbucnh-17","timestamp":"2021-03-12T02:33:02.751955799Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cPb986KctBPZNxu1j2usRVk","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.793972_117.1","section":"bucket.instance","name":"bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27","timestamp":"2021-03-12T02:33:02.793972919Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.799476_118.1","section":"bucket.instance","name":"bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27","timestamp":"2021-03-12T02:33:02.799476294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.801233_119.1","section":"bucket","name":"bbucnh-18","timestamp":"2021-03-12T02:33:02.801233293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bE0Z6UtgXyyVGQt7bxmNWss","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.805888_120.1","section":"bucket","name":"bbucnh-18","timestamp":"2021-03-12T02:33:02.805888319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bE0Z6UtgXyyVGQt7bxmNWss","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.833909_121.1","section":"bucket.instance","name":"bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25","timestamp":"2021-03-12T02:33:02.833909432Z","data":{"read_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":1},"write_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":2},"status":{"status":"write"}}},{"id":"1_1615516382.839228_122.1","section":"bucket.instance","name":"bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25","timestamp":"2021-03-12T02:33:02.839228126Z","data":{"read_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":1},"write_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516382.844254_123.1","section":"bucket.instance","name":"bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26","timestamp":"2021-03-12T02:33:02.844254956Z","data":{"read_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":1},"write_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":2},"status":{"status":"write"}}},{"id":"1_1615516382.850139_124.1","section":"bucket.instance","name":"bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26","timestamp":"2021-03-12T02:33:02.850139352Z","data":{"read_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":1},"write_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516382.860106_125.1","section":"bucket.instance","name":"bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27","timestamp":"2021-03-12T02:33:02.860106932Z","data":{"read_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":1},"write_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":2},"status":{"status":"write"}}},{"id":"1_1615516382.866387_126.1","section":"bucket.instance","name":"bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27","timestamp":"2021-03-12T02:33:02.866387331Z","data":{"read_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":1},"write_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516477.625386_179.1","section":"bucket.instance","name":"bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28","timestamp":"2021-03-12T02:34:37.625386901Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":1},"status":{"status":"write"}}},{"id":"1_1615516477.630575_180.1","section":"bucket.instance","name":"bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28","timestamp":"2021-03-12T02:34:37.630575238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516477.632196_181.1","section":"bucket","name":"bbucnh-19","timestamp":"2021-03-12T02:34:37.632196980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nPMLF28l579DSvKw8ZKyOBr","ver":1},"status":{"status":"write"}}},{"id":"1_1615516477.636638_182.1","section":"bucket","name":"bbucnh-19","timestamp":"2021-03-12T02:34:37.636638396Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nPMLF28l579DSvKw8ZKyOBr","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516477.641759_183.1","section":"bucket.instance","name":"bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28","timestamp":"2021-03-12T02:34:37.641759734Z","data":{"read_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":1},"write_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":2},"status":{"status":"write"}}},{"id":"1_1615516477.646256_184.1","section":"bucket.instance","name":"bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28","timestamp":"2021-03-12T02:34:37.646256599Z","data":{"read_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":1},"write_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516542.961983_188.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:35:42.961983629Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":1},"status":{"status":"write"}}},{"id":"1_1615516542.966262_189.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:35:42.966262443Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516542.968145_190.1","section":"bucket","name":"bbucnh-20","timestamp":"2021-03-12T02:35:42.968145462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jmYhKySMPNTjtHT_ULsBbuI","ver":1},"status":{"status":"write"}}},{"id":"1_1615516542.972050_191.1","section":"bucket","name":"bbucnh-20","timestamp":"2021-03-12T02:35:42.972050294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jmYhKySMPNTjtHT_ULsBbuI","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516560.741881_192.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:36:00.741881668Z","data":{"read_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":1},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":2},"status":{"status":"write"}}},{"id":"1_1615516560.750253_193.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:36:00.750253702Z","data":{"read_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":1},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516580.510392_194.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:36:20.510392140Z","data":{"read_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":2},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":3},"status":{"status":"write"}}},{"id":"1_1615516580.518628_195.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:36:20.518628079Z","data":{"read_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":2},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":3},"status":{"status":"complete"}}},{"id":"1_1615516601.288975_196.1","section":"bucket.instance","name":"bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30","timestamp":"2021-03-12T02:36:41.288975015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.293430_197.1","section":"bucket.instance","name":"bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30","timestamp":"2021-03-12T02:36:41.293430632Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.295149_198.1","section":"bucket","name":"bbucnh-21","timestamp":"2021-03-12T02:36:41.295149172Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__zd-Z1R4RWVmQc9qnWCZJ9Z","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.301563_199.1","section":"bucket","name":"bbucnh-21","timestamp":"2021-03-12T02:36:41.301563274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__zd-Z1R4RWVmQc9qnWCZJ9Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.326503_200.1","section":"bucket.instance","name":"bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31","timestamp":"2021-03-12T02:36:41.326503965Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.331867_201.1","section":"bucket.instance","name":"bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31","timestamp":"2021-03-12T02:36:41.331867729Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.333775_202.1","section":"bucket","name":"bbucnh-22","timestamp":"2021-03-12T02:36:41.333775173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lRVZbkb3M5VDTFzWzb2Ao_n","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.338365_203.1","section":"bucket","name":"bbucnh-22","timestamp":"2021-03-12T02:36:41.338365599Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lRVZbkb3M5VDTFzWzb2Ao_n","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.375262_204.1","section":"bucket.instance","name":"bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32","timestamp":"2021-03-12T02:36:41.375262696Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.380184_205.1","section":"bucket.instance","name":"bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32","timestamp":"2021-03-12T02:36:41.380184001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.382095_206.1","section":"bucket","name":"bbucnh-23","timestamp":"2021-03-12T02:36:41.382095207Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ox0KlvX6xNF4jdqzrljR_Yz","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.386448_207.1","section":"bucket","name":"bbucnh-23","timestamp":"2021-03-12T02:36:41.386448784Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ox0KlvX6xNF4jdqzrljR_Yz","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.406400_208.1","section":"bucket.instance","name":"bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30","timestamp":"2021-03-12T02:36:41.406400290Z","data":{"read_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":1},"write_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":2},"status":{"status":"write"}}},{"id":"1_1615516601.411130_209.1","section":"bucket.instance","name":"bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30","timestamp":"2021-03-12T02:36:41.411130339Z","data":{"read_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":1},"write_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516601.415296_210.1","section":"bucket.instance","name":"bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31","timestamp":"2021-03-12T02:36:41.415296943Z","data":{"read_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":1},"write_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":2},"status":{"status":"write"}}},{"id":"1_1615516601.420232_211.1","section":"bucket.instance","name":"bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31","timestamp":"2021-03-12T02:36:41.420232785Z","data":{"read_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":1},"write_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516601.429424_212.1","section":"bucket.instance","name":"bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32","timestamp":"2021-03-12T02:36:41.429424011Z","data":{"read_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":1},"write_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":2},"status":{"status":"write"}}},{"id":"1_1615516601.435061_213.1","section":"bucket.instance","name":"bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32","timestamp":"2021-03-12T02:36:41.435061595Z","data":{"read_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":1},"write_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516623.905134_226.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:03.905134670Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":1},"status":{"status":"write"}}},{"id":"1_1615516623.909582_227.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:03.909582485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516623.911054_228.1","section":"bucket","name":"bbucnh-24","timestamp":"2021-03-12T02:37:03.911054290Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kAh_Dud7mDE7666DS44bDVH","ver":1},"status":{"status":"write"}}},{"id":"1_1615516623.915168_229.1","section":"bucket","name":"bbucnh-24","timestamp":"2021-03-12T02:37:03.915168538Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kAh_Dud7mDE7666DS44bDVH","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516623.938779_230.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:03.938779082Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":1},"status":{"status":"write"}}},{"id":"1_1615516623.944057_231.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:03.944057002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516623.945869_232.1","section":"bucket","name":"bbucnh-25","timestamp":"2021-03-12T02:37:03.945869854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FVWFGEedKW_N26l2XXHS__P","ver":1},"status":{"status":"write"}}},{"id":"1_1615516623.952222_233.1","section":"bucket","name":"bbucnh-25","timestamp":"2021-03-12T02:37:03.952222510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FVWFGEedKW_N26l2XXHS__P","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516623.998369_234.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:03.998369746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":1},"status":{"status":"write"}}},{"id":"1_1615516624.003251_235.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.003251527Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516624.004804_236.1","section":"bucket","name":"bbucnh-26","timestamp":"2021-03-12T02:37:04.004804195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pZWbNId4gDq01c4QPLoSkT3","ver":1},"status":{"status":"write"}}},{"id":"1_1615516624.009867_237.1","section":"bucket","name":"bbucnh-26","timestamp":"2021-03-12T02:37:04.009867850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pZWbNId4gDq01c4QPLoSkT3","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516624.034129_238.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:04.034129873Z","data":{"read_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":1},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":2},"status":{"status":"write"}}},{"id":"1_1615516624.040808_239.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:04.040808023Z","data":{"read_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":1},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516624.044134_240.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:04.044134611Z","data":{"read_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":2},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":3},"status":{"status":"write"}}},{"id":"1_1615516624.051708_241.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:04.051708030Z","data":{"read_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":2},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":3},"status":{"status":"complete"}}},{"id":"1_1615516624.056887_242.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:04.056887996Z","data":{"read_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":1},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":2},"status":{"status":"write"}}},{"id":"1_1615516624.063220_243.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:04.063220593Z","data":{"read_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":1},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516624.073512_244.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:04.073512038Z","data":{"read_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":2},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":3},"status":{"status":"write"}}},{"id":"1_1615516624.080300_245.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:04.080300997Z","data":{"read_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":2},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":3},"status":{"status":"complete"}}},{"id":"1_1615516624.092046_246.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.092046934Z","data":{"read_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":1},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":2},"status":{"status":"write"}}},{"id":"1_1615516624.098396_247.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.098396048Z","data":{"read_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":1},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516624.109918_248.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.109918336Z","data":{"read_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":2},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":3},"status":{"status":"write"}}},{"id":"1_1615516624.119886_249.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.119886963Z","data":{"read_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":2},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":3},"status":{"status":"complete"}}},{"id":"1_1615516648.953955_260.1","section":"bucket.instance","name":"bbucnh-27:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.36","timestamp":"2021-03-12T02:37:28.953955696Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516648.958378_261.1","section":"bucket.instance","name":"bbucnh-27:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.36","timestamp":"2021-03-12T02:37:28.958378293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516648.959965_262.1","section":"bucket","name":"bbucnh-27","timestamp":"2021-03-12T02:37:28.959965688Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FmfQYONhZDXgO1uw7EYs9_S","ver":1},"status":{"status":"write"}}},{"id":"1_1615516648.963902_263.1","section":"bucket","name":"bbucnh-27","timestamp":"2021-03-12T02:37:28.963902711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FmfQYONhZDXgO1uw7EYs9_S","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516648.987286_264.1","section":"bucket.instance","name":"bbucnh-28:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37","timestamp":"2021-03-12T02:37:28.987286166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":1},"status":{"status":"write"}}},{"id":"1_1615516648.993373_265.1","section":"bucket.instance","name":"bbucnh-28:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37","timestamp":"2021-03-12T02:37:28.993373542Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516648.995310_266.1","section":"bucket","name":"bbucnh-28","timestamp":"2021-03-12T02:37:28.995310434Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2gVXe3dHFNHJj_5mBzi7BFQ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.000050_267.1","section":"bucket","name":"bbucnh-28","timestamp":"2021-03-12T02:37:29.000050799Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2gVXe3dHFNHJj_5mBzi7BFQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.040216_268.1","section":"bucket.instance","name":"bbucnh-29:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38","timestamp":"2021-03-12T02:37:29.040216016Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.045685_269.1","section":"bucket.instance","name":"bbucnh-29:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38","timestamp":"2021-03-12T02:37:29.045685128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.047750_270.1","section":"bucket","name":"bbucnh-29","timestamp":"2021-03-12T02:37:29.047750134Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NzPM--eEDeYlpvhbh0MdeK4","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.052200_271.1","section":"bucket","name":"bbucnh-29","timestamp":"2021-03-12T02:37:29.052200209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NzPM--eEDeYlpvhbh0MdeK4","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.075303_272.1","section":"bucket.instance","name":"bbucnh-27:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.36","timestamp":"2021-03-12T02:37:29.075303587Z","data":{"read_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":1},"write_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.080878_273.1","section":"bucket.instance","name":"bbucnh-27:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.36","timestamp":"2021-03-12T02:37:29.080878607Z","data":{"read_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":1},"write_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.129020_274.1","section":"bucket.instance","name":"bbucnh-28:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37","timestamp":"2021-03-12T02:37:29.129020143Z","data":{"read_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":1},"write_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.137207_275.1","section":"bucket.instance","name":"bbucnh-28:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37","timestamp":"2021-03-12T02:37:29.137207384Z","data":{"read_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":1},"write_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.196723_276.1","section":"bucket.instance","name":"bbucnh-29:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38","timestamp":"2021-03-12T02:37:29.196723737Z","data":{"read_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":1},"write_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.204941_277.1","section":"bucket.instance","name":"bbucnh-29:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38","timestamp":"2021-03-12T02:37:29.204941290Z","data":{"read_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":1},"write_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.273026_278.1","section":"bucket.instance","name":"bbucnh-30:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.39","timestamp":"2021-03-12T02:37:29.273026550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.278145_279.1","section":"bucket.instance","name":"bbucnh-30:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.39","timestamp":"2021-03-12T02:37:29.278145289Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.279975_280.1","section":"bucket","name":"bbucnh-30","timestamp":"2021-03-12T02:37:29.279975374Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ktkorz1HBIgiOiSGUck37Fi","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.284871_281.1","section":"bucket","name":"bbucnh-30","timestamp":"2021-03-12T02:37:29.284871749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ktkorz1HBIgiOiSGUck37Fi","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.312142_282.1","section":"bucket.instance","name":"bbucnh-31:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40","timestamp":"2021-03-12T02:37:29.312142460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.317109_283.1","section":"bucket.instance","name":"bbucnh-31:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40","timestamp":"2021-03-12T02:37:29.317109530Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.318899_284.1","section":"bucket","name":"bbucnh-31","timestamp":"2021-03-12T02:37:29.318899628Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RVIUeIv4NvsRyQ3Z9VfRls-","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.323537_285.1","section":"bucket","name":"bbucnh-31","timestamp":"2021-03-12T02:37:29.323537422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RVIUeIv4NvsRyQ3Z9VfRls-","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.364204_286.1","section":"bucket.instance","name":"bbucnh-32:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41","timestamp":"2021-03-12T02:37:29.364204861Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.369006_287.1","section":"bucket.instance","name":"bbucnh-32:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41","timestamp":"2021-03-12T02:37:29.369006270Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.370821_288.1","section":"bucket","name":"bbucnh-32","timestamp":"2021-03-12T02:37:29.370821226Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uNNdkr5c4kYGRpyB0KL4L6X","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.374864_289.1","section":"bucket","name":"bbucnh-32","timestamp":"2021-03-12T02:37:29.374864824Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uNNdkr5c4kYGRpyB0KL4L6X","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.400473_291.1","section":"bucket.instance","name":"bbucnh-30:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.39","timestamp":"2021-03-12T02:37:29.400473777Z","data":{"read_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":1},"write_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.405529_292.1","section":"bucket.instance","name":"bbucnh-30:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.39","timestamp":"2021-03-12T02:37:29.405529038Z","data":{"read_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":1},"write_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.454396_293.1","section":"bucket.instance","name":"bbucnh-31:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40","timestamp":"2021-03-12T02:37:29.454396735Z","data":{"read_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":1},"write_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.462159_294.1","section":"bucket.instance","name":"bbucnh-31:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40","timestamp":"2021-03-12T02:37:29.462159544Z","data":{"read_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":1},"write_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.526776_295.1","section":"bucket.instance","name":"bbucnh-32:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41","timestamp":"2021-03-12T02:37:29.526776297Z","data":{"read_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":1},"write_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.534441_296.1","section":"bucket.instance","name":"bbucnh-32:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41","timestamp":"2021-03-12T02:37:29.534441885Z","data":{"read_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":1},"write_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.609051_298.1","section":"bucket.instance","name":"bbucnh-33:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.42","timestamp":"2021-03-12T02:37:29.609051947Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.613737_299.1","section":"bucket.instance","name":"bbucnh-33:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.42","timestamp":"2021-03-12T02:37:29.613737442Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.615385_300.1","section":"bucket","name":"bbucnh-33","timestamp":"2021-03-12T02:37:29.615385271Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_15Kd9p4ie5l3fyoCESSGjmB","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.619356_301.1","section":"bucket","name":"bbucnh-33","timestamp":"2021-03-12T02:37:29.619356209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_15Kd9p4ie5l3fyoCESSGjmB","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.644293_302.1","section":"bucket.instance","name":"bbucnh-34:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43","timestamp":"2021-03-12T02:37:29.644293956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.649652_303.1","section":"bucket.instance","name":"bbucnh-34:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43","timestamp":"2021-03-12T02:37:29.649652088Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.651633_304.1","section":"bucket","name":"bbucnh-34","timestamp":"2021-03-12T02:37:29.651633615Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KxY1ghborztF7ijmoRBmfYR","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.656854_305.1","section":"bucket","name":"bbucnh-34","timestamp":"2021-03-12T02:37:29.656854451Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KxY1ghborztF7ijmoRBmfYR","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.700347_306.1","section":"bucket.instance","name":"bbucnh-35:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44","timestamp":"2021-03-12T02:37:29.700347440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.705720_307.1","section":"bucket.instance","name":"bbucnh-35:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44","timestamp":"2021-03-12T02:37:29.705720561Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.707370_308.1","section":"bucket","name":"bbucnh-35","timestamp":"2021-03-12T02:37:29.707370179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tJebU3psH_SbzeVFZfWinUh","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.711408_309.1","section":"bucket","name":"bbucnh-35","timestamp":"2021-03-12T02:37:29.711408051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tJebU3psH_SbzeVFZfWinUh","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.734891_310.1","section":"bucket.instance","name":"bbucnh-33:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.42","timestamp":"2021-03-12T02:37:29.734891456Z","data":{"read_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":1},"write_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.739920_311.1","section":"bucket.instance","name":"bbucnh-33:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.42","timestamp":"2021-03-12T02:37:29.739920185Z","data":{"read_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":1},"write_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516650.789737_312.1","section":"bucket.instance","name":"bbucnh-34:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43","timestamp":"2021-03-12T02:37:30.789737190Z","data":{"read_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":1},"write_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":2},"status":{"status":"write"}}},{"id":"1_1615516650.798206_313.1","section":"bucket.instance","name":"bbucnh-34:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43","timestamp":"2021-03-12T02:37:30.798206828Z","data":{"read_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":1},"write_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516652.335363_314.1","section":"bucket.instance","name":"bbucnh-35:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44","timestamp":"2021-03-12T02:37:32.335363586Z","data":{"read_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":1},"write_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":2},"status":{"status":"write"}}},{"id":"1_1615516652.344811_315.1","section":"bucket.instance","name":"bbucnh-35:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44","timestamp":"2021-03-12T02:37:32.344811119Z","data":{"read_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":1},"write_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516653.069940_316.1","section":"bucket.instance","name":"bbucnh-36:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45","timestamp":"2021-03-12T02:37:33.069940667Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VuDCjriGUmI_jGsCkSwyMPM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516653.075083_317.1","section":"bucket.instance","name":"bbucnh-36:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45","timestamp":"2021-03-12T02:37:33.075083097Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VuDCjriGUmI_jGsCkSwyMPM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516653.076710_318.1","section":"bucket","name":"bbucnh-36","timestamp":"2021-03-12T02:37:33.076710866Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6lGSmQvlrfayDP125PAmHZF","ver":1},"status":{"status":"write"}}},{"id":"1_1615516653.081538_319.1","section":"bucket","name":"bbucnh-36","timestamp":"2021-03-12T02:37:33.081538100Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6lGSmQvlrfayDP125PAmHZF","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516653.664443_320.1","section":"bucket.instance","name":"bbucnh-37:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46","timestamp":"2021-03-12T02:37:33.664443824Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HmfgDDWFN0bOGRAMO2ByNk7","ver":1},"status":{"status":"write"}}},{"id":"1_1615516653.669629_321.1","section":"bucket.instance","name":"bbucnh-37:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46","timestamp":"2021-03-12T02:37:33.669629721Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HmfgDDWFN0bOGRAMO2ByNk7","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516653.671304_322.1","section":"bucket","name":"bbucnh-37","timestamp":"2021-03-12T02:37:33.671304239Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J6fc8NV-QkiOt2H3vUtcdSS","ver":1},"status":{"status":"write"}}},{"id":"1_1615516653.675804_323.1","section":"bucket","name":"bbucnh-37","timestamp":"2021-03-12T02:37:33.675804856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J6fc8NV-QkiOt2H3vUtcdSS","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516653.998533_324.1","section":"bucket.instance","name":"bbucnh-38:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47","timestamp":"2021-03-12T02:37:33.998533211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DXFrPBq7cR1YlfkiP9rTC_h","ver":1},"status":{"status":"write"}}},{"id":"1_1615516654.002986_325.1","section":"bucket.instance","name":"bbucnh-38:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47","timestamp":"2021-03-12T02:37:34.002986261Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DXFrPBq7cR1YlfkiP9rTC_h","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516654.004526_326.1","section":"bucket","name":"bbucnh-38","timestamp":"2021-03-12T02:37:34.004526175Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B2U5VzBNff39d-uvhTwPTVA","ver":1},"status":{"status":"write"}}},{"id":"1_1615516654.008644_327.1","section":"bucket","name":"bbucnh-38","timestamp":"2021-03-12T02:37:34.008644344Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B2U5VzBNff39d-uvhTwPTVA","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.472251_328.1","section":"bucket.instance","name":"bbucnh-39:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.48","timestamp":"2021-03-12T02:37:36.472251903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxwXPsBw5MBOuLy2DL0RzSC","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.477307_329.1","section":"bucket.instance","name":"bbucnh-39:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.48","timestamp":"2021-03-12T02:37:36.477307238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxwXPsBw5MBOuLy2DL0RzSC","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.479321_330.1","section":"bucket","name":"bbucnh-39","timestamp":"2021-03-12T02:37:36.479321851Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7RmOoO6yS0MAc_ELYlYq8yN","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.483505_331.1","section":"bucket","name":"bbucnh-39","timestamp":"2021-03-12T02:37:36.483505123Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7RmOoO6yS0MAc_ELYlYq8yN","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.511042_332.1","section":"bucket.instance","name":"bbucnh-40:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.49","timestamp":"2021-03-12T02:37:36.511042128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q1cAgCTgGfyj4SRitg3YHYj","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.516504_333.1","section":"bucket.instance","name":"bbucnh-40:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.49","timestamp":"2021-03-12T02:37:36.516504986Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q1cAgCTgGfyj4SRitg3YHYj","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.518471_334.1","section":"bucket","name":"bbucnh-40","timestamp":"2021-03-12T02:37:36.518471862Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vmcO4AQPJh3F7zPA8Q1MDcO","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.522955_335.1","section":"bucket","name":"bbucnh-40","timestamp":"2021-03-12T02:37:36.522955683Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vmcO4AQPJh3F7zPA8Q1MDcO","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.569936_336.1","section":"bucket.instance","name":"bbucnh-41:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.50","timestamp":"2021-03-12T02:37:36.569936921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V9oej-UfPhBNN29T2LbubWl","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.574506_337.1","section":"bucket.instance","name":"bbucnh-41:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.50","timestamp":"2021-03-12T02:37:36.574506107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V9oej-UfPhBNN29T2LbubWl","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.576240_338.1","section":"bucket","name":"bbucnh-41","timestamp":"2021-03-12T02:37:36.576240547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TmSKoQq5CYyZMuTTOWMF8jB","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.580426_339.1","section":"bucket","name":"bbucnh-41","timestamp":"2021-03-12T02:37:36.580426570Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TmSKoQq5CYyZMuTTOWMF8jB","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516277.246267_81.1","section":"bucket.instance","name":"bbucnh-1:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.1","timestamp":"2021-03-12T02:31:17.246267120Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qfxaS5zdrJis58Zg6Obg75g","ver":1},"status":{"status":"write"}}},{"id":"1_1615516277.255262_85.1","section":"bucket.instance","name":"bbucnh-1:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.1","timestamp":"2021-03-12T02:31:17.255262459Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qfxaS5zdrJis58Zg6Obg75g","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516277.256664_86.1","section":"bucket","name":"bbucnh-1","timestamp":"2021-03-12T02:31:17.256664887Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SJQ8Ox0nk_KuM-tokbc3CHJ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516277.260851_87.1","section":"bucket","name":"bbucnh-1","timestamp":"2021-03-12T02:31:17.260851736Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SJQ8Ox0nk_KuM-tokbc3CHJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516277.279301_88.1","section":"bucket.instance","name":"bbucnh-2:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.2","timestamp":"2021-03-12T02:31:17.279301358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OdmYkD6xh5KguUnb-mdZ1aM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516277.284169_89.1","section":"bucket.instance","name":"bbucnh-2:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.2","timestamp":"2021-03-12T02:31:17.284169621Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OdmYkD6xh5KguUnb-mdZ1aM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516277.285818_90.1","section":"bucket","name":"bbucnh-2","timestamp":"2021-03-12T02:31:17.285818007Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U61zBQN8Y4-IaOqXqUfnFDK","ver":1},"status":{"status":"write"}}},{"id":"1_1615516277.289902_91.1","section":"bucket","name":"bbucnh-2","timestamp":"2021-03-12T02:31:17.289902101Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U61zBQN8Y4-IaOqXqUfnFDK","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516280.582981_92.1","section":"bucket.instance","name":"bbucnh-3:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.3","timestamp":"2021-03-12T02:31:20.582981563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WmExOXe_DXwUuw8ZWwStmCG","ver":1},"status":{"status":"write"}}},{"id":"1_1615516280.587282_93.1","section":"bucket.instance","name":"bbucnh-3:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.3","timestamp":"2021-03-12T02:31:20.587282731Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WmExOXe_DXwUuw8ZWwStmCG","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516280.588897_94.1","section":"bucket","name":"bbucnh-3","timestamp":"2021-03-12T02:31:20.588897050Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vTlspttjgtqvLuUNPCk_X1Z","ver":1},"status":{"status":"write"}}},{"id":"1_1615516280.593132_95.1","section":"bucket","name":"bbucnh-3","timestamp":"2021-03-12T02:31:20.593132328Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vTlspttjgtqvLuUNPCk_X1Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516289.394488_96.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.4","timestamp":"2021-03-12T02:31:29.394488830Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P9XE_bWiNIw3vvd5uTEfxr3","ver":1},"status":{"status":"write"}}},{"id":"1_1615516289.399511_98.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.4","timestamp":"2021-03-12T02:31:29.399511189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P9XE_bWiNIw3vvd5uTEfxr3","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516289.400904_99.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:29.400904315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HJA5lWRMWyY0IaqyBfFi1N4","ver":1},"status":{"status":"write"}}},{"id":"1_1615516289.404420_100.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:29.404420265Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HJA5lWRMWyY0IaqyBfFi1N4","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516289.903936_101.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.5","timestamp":"2021-03-12T02:31:29.903936280Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1ecwCnmWKxBMRrpYQmvXMe-","ver":1},"status":{"status":"write"}}},{"id":"1_1615516289.908750_102.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.5","timestamp":"2021-03-12T02:31:29.908750593Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1ecwCnmWKxBMRrpYQmvXMe-","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516289.910808_103.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:29.910808743Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nZiOpvQgZemMdcUiOXIn1VT","ver":1},"status":{"status":"write"}}},{"id":"1_1615516289.916752_104.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:29.916752691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nZiOpvQgZemMdcUiOXIn1VT","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516290.449434_105.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.6","timestamp":"2021-03-12T02:31:30.449434628Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rHxZdUR046Wgm3Zafw0Xi8l","ver":1},"status":{"status":"write"}}},{"id":"1_1615516290.453654_106.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.6","timestamp":"2021-03-12T02:31:30.453654797Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rHxZdUR046Wgm3Zafw0Xi8l","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516290.455139_107.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:30.455139379Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sg78H-Zgxy2C6QwruYgc-Rg","ver":1},"status":{"status":"write"}}},{"id":"1_1615516290.459112_108.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:30.459112128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sg78H-Zgxy2C6QwruYgc-Rg","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.676123_109.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.7","timestamp":"2021-03-12T02:31:32.676123449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T4BoTd-m-PH42nJonMN7kZP","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.681307_111.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.7","timestamp":"2021-03-12T02:31:32.681307323Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T4BoTd-m-PH42nJonMN7kZP","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.683099_112.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:32.683099550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EBuzwkBweLN5rvCeTKt4REO","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.688466_113.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:32.688466865Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EBuzwkBweLN5rvCeTKt4REO","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.712314_114.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.8","timestamp":"2021-03-12T02:31:32.712314715Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c8czZW2QuWkq0wJE4l-Nvwz","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.716801_115.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.8","timestamp":"2021-03-12T02:31:32.716801906Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c8czZW2QuWkq0wJE4l-Nvwz","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.718398_116.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:32.718398180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yjAubSLjTLPIuEzKEWdFJD8","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.721131_117.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:32.721131522Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yjAubSLjTLPIuEzKEWdFJD8","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.742288_118.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.9","timestamp":"2021-03-12T02:31:32.742288024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cd_ZmHPW-yz7g9wMfu8z1OM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.746471_119.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.9","timestamp":"2021-03-12T02:31:32.746471019Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cd_ZmHPW-yz7g9wMfu8z1OM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.748067_120.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:32.748067528Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yyCMzqqMnUhRq-wn-e0Oe_1","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.752347_121.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:32.752347222Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yyCMzqqMnUhRq-wn-e0Oe_1","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.782913_122.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.10","timestamp":"2021-03-12T02:31:32.782913622Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cbSranN-_o_cTrR-FcWF_2Y","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.788130_123.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.10","timestamp":"2021-03-12T02:31:32.788130810Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cbSranN-_o_cTrR-FcWF_2Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.790158_124.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:32.790158153Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7PSv7RO8pR2twgmTFBWqx0N","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.795541_125.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:32.795541882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7PSv7RO8pR2twgmTFBWqx0N","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.844603_126.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.11","timestamp":"2021-03-12T02:31:32.844603751Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4_B8b59PwgOUx0apcPz4YE3","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.849848_127.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.11","timestamp":"2021-03-12T02:31:32.849848924Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4_B8b59PwgOUx0apcPz4YE3","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.851664_128.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:32.851664510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G_5ocbuuJ8e81M9Oey3KkUa","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.855275_129.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:32.855275037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G_5ocbuuJ8e81M9Oey3KkUa","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.947746_130.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.12","timestamp":"2021-03-12T02:31:32.947746750Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hLAW_-1vXKVBgOh8Z3yp4RR","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.952705_131.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.12","timestamp":"2021-03-12T02:31:32.952705876Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hLAW_-1vXKVBgOh8Z3yp4RR","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.954374_132.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:32.954374195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u-RVInQ17b5MGi2bJAMXC9c","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.957220_133.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:32.957220699Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u-RVInQ17b5MGi2bJAMXC9c","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516293.053188_134.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.13","timestamp":"2021-03-12T02:31:33.053188210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3Sm3iR_O9gOws1Qm0XHBesI","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.059103_135.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.13","timestamp":"2021-03-12T02:31:33.059103540Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3Sm3iR_O9gOws1Qm0XHBesI","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516293.060699_136.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:33.060699635Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fJHHqFsf0ZnQ8HA3eVv0HsQ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.063543_137.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:33.063543264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fJHHqFsf0ZnQ8HA3eVv0HsQ","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516293.100428_138.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.14","timestamp":"2021-03-12T02:31:33.100428411Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qEcq5zwRQGvPWI89_dceYVx","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.105635_139.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.14","timestamp":"2021-03-12T02:31:33.105635408Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qEcq5zwRQGvPWI89_dceYVx","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516293.107322_140.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:33.107322438Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_usXxiQpihhvHv9w9H2KqHQL","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.112787_141.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:33.112787593Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_usXxiQpihhvHv9w9H2KqHQL","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516293.205051_142.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.15","timestamp":"2021-03-12T02:31:33.205051135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NyLEienNgXaYmXOscru4QK2","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.210392_143.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.15","timestamp":"2021-03-12T02:31:33.210392899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NyLEienNgXaYmXOscru4QK2","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516293.212220_144.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:33.212220343Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_McGu-bUTB7Iwrzro3KX9U6v","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.216631_145.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:33.216631468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_McGu-bUTB7Iwrzro3KX9U6v","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516300.493202_146.1","section":"bucket.instance","name":"bbucnh-7:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.16","timestamp":"2021-03-12T02:31:40.493202575Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJvoXXdfbZSbk6J8UPJzIDB","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.498785_148.1","section":"bucket.instance","name":"bbucnh-7:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.16","timestamp":"2021-03-12T02:31:40.498785950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJvoXXdfbZSbk6J8UPJzIDB","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.500553_149.1","section":"bucket","name":"bbucnh-7","timestamp":"2021-03-12T02:31:40.500553657Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.507000_150.1","section":"bucket","name":"bbucnh-7","timestamp":"2021-03-12T02:31:40.507000938Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.528275_152.1","section":"bucket.instance","name":"bbucnh-8:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.17","timestamp":"2021-03-12T02:31:40.528275074Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n22OShRJQ_FxFcZr65eyDbq","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.533734_153.1","section":"bucket.instance","name":"bbucnh-8:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.17","timestamp":"2021-03-12T02:31:40.533734931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n22OShRJQ_FxFcZr65eyDbq","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.535436_154.1","section":"bucket","name":"bbucnh-8","timestamp":"2021-03-12T02:31:40.535436833Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.541545_155.1","section":"bucket","name":"bbucnh-8","timestamp":"2021-03-12T02:31:40.541545383Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.583042_156.1","section":"bucket.instance","name":"bbucnh-9:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.18","timestamp":"2021-03-12T02:31:40.583042072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WFleci_lBAEwlmYvNA5F_z0","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.587676_157.1","section":"bucket.instance","name":"bbucnh-9:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.18","timestamp":"2021-03-12T02:31:40.587676557Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WFleci_lBAEwlmYvNA5F_z0","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.589578_158.1","section":"bucket","name":"bbucnh-9","timestamp":"2021-03-12T02:31:40.589578574Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.595034_159.1","section":"bucket","name":"bbucnh-9","timestamp":"2021-03-12T02:31:40.595034275Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516309.326188_162.1","section":"bucket","name":"bbucnh-7","timestamp":"2021-03-12T02:31:49.326188604Z","data":{"read_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":1},"write_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":2},"status":{"status":"remove"}}},{"id":"1_1615516309.332337_163.1","section":"bucket","name":"bbucnh-7","timestamp":"2021-03-12T02:31:49.332337164Z","data":{"read_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":1},"write_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516309.508827_165.1","section":"bucket","name":"bbucnh-8","timestamp":"2021-03-12T02:31:49.508827513Z","data":{"read_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":1},"write_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":2},"status":{"status":"remove"}}},{"id":"1_1615516309.514944_166.1","section":"bucket","name":"bbucnh-8","timestamp":"2021-03-12T02:31:49.514944882Z","data":{"read_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":1},"write_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516309.592520_167.1","section":"bucket","name":"bbucnh-9","timestamp":"2021-03-12T02:31:49.592520057Z","data":{"read_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":1},"write_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":2},"status":{"status":"remove"}}},{"id":"1_1615516309.597935_168.1","section":"bucket","name":"bbucnh-9","timestamp":"2021-03-12T02:31:49.597935909Z","data":{"read_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":1},"write_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":2},"status":{"status":"complete"}}}] 2021-03-12T02:40:10.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0cf3ee11-29e6-4776-941b-50bda62adf82', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:10.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0cf3ee11-29e6-4776-941b-50bda62adf82', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:10.731 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0cf3ee11-29e6-4776-941b-50bda62adf82 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:10.775 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:10.776 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:10.992 INFO:teuthology.orchestra.run.smithi114.stdout:[{"id":"1_1615516156.388233_96.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-12T02:29:16.388233974Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YL3_4pFMNCff-5Idsb0xw0Y","ver":1},"status":{"status":"write"}}},{"id":"1_1615516156.395861_98.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-12T02:29:16.395861436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YL3_4pFMNCff-5Idsb0xw0Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.517635_58.1","section":"bucket.instance","name":"bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19","timestamp":"2021-03-12T02:31:57.517635735Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jcLKJHBt6Ev7gDksP2Slh3b","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.523069_59.1","section":"bucket.instance","name":"bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19","timestamp":"2021-03-12T02:31:57.523069746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jcLKJHBt6Ev7gDksP2Slh3b","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.525015_60.1","section":"bucket","name":"bbucnh-10","timestamp":"2021-03-12T02:31:57.525015146Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6lPftr6jYyKA9E3-g-_Jn6a","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.529693_61.1","section":"bucket","name":"bbucnh-10","timestamp":"2021-03-12T02:31:57.529693741Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6lPftr6jYyKA9E3-g-_Jn6a","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.924299_62.1","section":"bucket.instance","name":"bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20","timestamp":"2021-03-12T02:31:57.924299109Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J0Dn8QzeuUr9s9ivyu7dsOD","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.930629_63.1","section":"bucket.instance","name":"bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20","timestamp":"2021-03-12T02:31:57.930629043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J0Dn8QzeuUr9s9ivyu7dsOD","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.932410_64.1","section":"bucket","name":"bbucnh-11","timestamp":"2021-03-12T02:31:57.932410309Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zofeXVzfWAE58gjk_kRUwx9","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.936714_65.1","section":"bucket","name":"bbucnh-11","timestamp":"2021-03-12T02:31:57.936714991Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zofeXVzfWAE58gjk_kRUwx9","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.973905_66.1","section":"bucket.instance","name":"bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21","timestamp":"2021-03-12T02:31:57.973905477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_25ydPCfdlzXA1KPDlkW_pFL","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.978105_67.1","section":"bucket.instance","name":"bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21","timestamp":"2021-03-12T02:31:57.978105914Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_25ydPCfdlzXA1KPDlkW_pFL","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516317.979768_68.1","section":"bucket","name":"bbucnh-12","timestamp":"2021-03-12T02:31:57.979768274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dw8z_OfXq9g-1vCkmj9nOKx","ver":1},"status":{"status":"write"}}},{"id":"1_1615516317.983732_69.1","section":"bucket","name":"bbucnh-12","timestamp":"2021-03-12T02:31:57.983732995Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dw8z_OfXq9g-1vCkmj9nOKx","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516335.993721_86.1","section":"bucket.instance","name":"bbucnh-13:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22","timestamp":"2021-03-12T02:32:15.993721453Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iFsto3hGG7H5MJRUgU2zSGi","ver":1},"status":{"status":"write"}}},{"id":"1_1615516335.999815_87.1","section":"bucket.instance","name":"bbucnh-13:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22","timestamp":"2021-03-12T02:32:15.999815759Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iFsto3hGG7H5MJRUgU2zSGi","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.001721_88.1","section":"bucket","name":"bbucnh-13","timestamp":"2021-03-12T02:32:16.001721436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dy_6thslFFy5DEBDJ7AdBBZ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.006149_89.1","section":"bucket","name":"bbucnh-13","timestamp":"2021-03-12T02:32:16.006149764Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dy_6thslFFy5DEBDJ7AdBBZ","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.023548_90.1","section":"bucket.instance","name":"bbucnh-14:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23","timestamp":"2021-03-12T02:32:16.023548341Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UidaAgtGQcz4suSP3NCWLHf","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.028425_91.1","section":"bucket.instance","name":"bbucnh-14:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23","timestamp":"2021-03-12T02:32:16.028425620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UidaAgtGQcz4suSP3NCWLHf","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.030253_92.1","section":"bucket","name":"bbucnh-14","timestamp":"2021-03-12T02:32:16.030253029Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8u_rE6TyERk7cInsKIFxC_W","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.034867_93.1","section":"bucket","name":"bbucnh-14","timestamp":"2021-03-12T02:32:16.034867278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8u_rE6TyERk7cInsKIFxC_W","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.073203_94.1","section":"bucket.instance","name":"bbucnh-15:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24","timestamp":"2021-03-12T02:32:16.073203702Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fPI6vD6iqij_2lOipNCA5Pj","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.078448_95.1","section":"bucket.instance","name":"bbucnh-15:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24","timestamp":"2021-03-12T02:32:16.078448093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fPI6vD6iqij_2lOipNCA5Pj","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516336.079985_96.1","section":"bucket","name":"bbucnh-15","timestamp":"2021-03-12T02:32:16.079985141Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZURiB1MDtruSPPRLjEHfFQn","ver":1},"status":{"status":"write"}}},{"id":"1_1615516336.084220_97.1","section":"bucket","name":"bbucnh-15","timestamp":"2021-03-12T02:32:16.084220471Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZURiB1MDtruSPPRLjEHfFQn","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.698485_109.1","section":"bucket.instance","name":"bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25","timestamp":"2021-03-12T02:33:02.698485717Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.704669_110.1","section":"bucket.instance","name":"bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25","timestamp":"2021-03-12T02:33:02.704669333Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.706515_111.1","section":"bucket","name":"bbucnh-16","timestamp":"2021-03-12T02:33:02.706515387Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3i1dwkEn1BGUdDnt5jjMXCT","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.713003_112.1","section":"bucket","name":"bbucnh-16","timestamp":"2021-03-12T02:33:02.713003405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3i1dwkEn1BGUdDnt5jjMXCT","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.737854_113.1","section":"bucket.instance","name":"bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26","timestamp":"2021-03-12T02:33:02.737854190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.743984_114.1","section":"bucket.instance","name":"bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26","timestamp":"2021-03-12T02:33:02.743984571Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.745895_115.1","section":"bucket","name":"bbucnh-17","timestamp":"2021-03-12T02:33:02.745895289Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cPb986KctBPZNxu1j2usRVk","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.751955_116.1","section":"bucket","name":"bbucnh-17","timestamp":"2021-03-12T02:33:02.751955799Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cPb986KctBPZNxu1j2usRVk","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.793972_117.1","section":"bucket.instance","name":"bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27","timestamp":"2021-03-12T02:33:02.793972919Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.799476_118.1","section":"bucket.instance","name":"bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27","timestamp":"2021-03-12T02:33:02.799476294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.801233_119.1","section":"bucket","name":"bbucnh-18","timestamp":"2021-03-12T02:33:02.801233293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bE0Z6UtgXyyVGQt7bxmNWss","ver":1},"status":{"status":"write"}}},{"id":"1_1615516382.805888_120.1","section":"bucket","name":"bbucnh-18","timestamp":"2021-03-12T02:33:02.805888319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bE0Z6UtgXyyVGQt7bxmNWss","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516382.833909_121.1","section":"bucket.instance","name":"bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25","timestamp":"2021-03-12T02:33:02.833909432Z","data":{"read_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":1},"write_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":2},"status":{"status":"write"}}},{"id":"1_1615516382.839228_122.1","section":"bucket.instance","name":"bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25","timestamp":"2021-03-12T02:33:02.839228126Z","data":{"read_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":1},"write_version":{"tag":"_mWQmb8LujTcAVgCjlcFTqdE","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516382.844254_123.1","section":"bucket.instance","name":"bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26","timestamp":"2021-03-12T02:33:02.844254956Z","data":{"read_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":1},"write_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":2},"status":{"status":"write"}}},{"id":"1_1615516382.850139_124.1","section":"bucket.instance","name":"bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26","timestamp":"2021-03-12T02:33:02.850139352Z","data":{"read_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":1},"write_version":{"tag":"_XFaB8RQxqvAsOe4NsnU4eGE","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516382.860106_125.1","section":"bucket.instance","name":"bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27","timestamp":"2021-03-12T02:33:02.860106932Z","data":{"read_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":1},"write_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":2},"status":{"status":"write"}}},{"id":"1_1615516382.866387_126.1","section":"bucket.instance","name":"bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27","timestamp":"2021-03-12T02:33:02.866387331Z","data":{"read_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":1},"write_version":{"tag":"_UADcOzQL_p25evhUbY0Bpmc","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516477.625386_179.1","section":"bucket.instance","name":"bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28","timestamp":"2021-03-12T02:34:37.625386901Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":1},"status":{"status":"write"}}},{"id":"1_1615516477.630575_180.1","section":"bucket.instance","name":"bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28","timestamp":"2021-03-12T02:34:37.630575238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516477.632196_181.1","section":"bucket","name":"bbucnh-19","timestamp":"2021-03-12T02:34:37.632196980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nPMLF28l579DSvKw8ZKyOBr","ver":1},"status":{"status":"write"}}},{"id":"1_1615516477.636638_182.1","section":"bucket","name":"bbucnh-19","timestamp":"2021-03-12T02:34:37.636638396Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nPMLF28l579DSvKw8ZKyOBr","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516477.641759_183.1","section":"bucket.instance","name":"bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28","timestamp":"2021-03-12T02:34:37.641759734Z","data":{"read_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":1},"write_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":2},"status":{"status":"write"}}},{"id":"1_1615516477.646256_184.1","section":"bucket.instance","name":"bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28","timestamp":"2021-03-12T02:34:37.646256599Z","data":{"read_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":1},"write_version":{"tag":"_nNtumYncIAYrLzBcHKJumSf","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516542.961983_188.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:35:42.961983629Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":1},"status":{"status":"write"}}},{"id":"1_1615516542.966262_189.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:35:42.966262443Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516542.968145_190.1","section":"bucket","name":"bbucnh-20","timestamp":"2021-03-12T02:35:42.968145462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jmYhKySMPNTjtHT_ULsBbuI","ver":1},"status":{"status":"write"}}},{"id":"1_1615516542.972050_191.1","section":"bucket","name":"bbucnh-20","timestamp":"2021-03-12T02:35:42.972050294Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jmYhKySMPNTjtHT_ULsBbuI","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516560.741881_192.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:36:00.741881668Z","data":{"read_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":1},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":2},"status":{"status":"write"}}},{"id":"1_1615516560.750253_193.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:36:00.750253702Z","data":{"read_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":1},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516580.510392_194.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:36:20.510392140Z","data":{"read_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":2},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":3},"status":{"status":"write"}}},{"id":"1_1615516580.518628_195.1","section":"bucket.instance","name":"bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29","timestamp":"2021-03-12T02:36:20.518628079Z","data":{"read_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":2},"write_version":{"tag":"_cBPTd9ThsDt13fUvbYdLRql","ver":3},"status":{"status":"complete"}}},{"id":"1_1615516601.288975_196.1","section":"bucket.instance","name":"bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30","timestamp":"2021-03-12T02:36:41.288975015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.293430_197.1","section":"bucket.instance","name":"bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30","timestamp":"2021-03-12T02:36:41.293430632Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.295149_198.1","section":"bucket","name":"bbucnh-21","timestamp":"2021-03-12T02:36:41.295149172Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__zd-Z1R4RWVmQc9qnWCZJ9Z","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.301563_199.1","section":"bucket","name":"bbucnh-21","timestamp":"2021-03-12T02:36:41.301563274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__zd-Z1R4RWVmQc9qnWCZJ9Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.326503_200.1","section":"bucket.instance","name":"bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31","timestamp":"2021-03-12T02:36:41.326503965Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.331867_201.1","section":"bucket.instance","name":"bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31","timestamp":"2021-03-12T02:36:41.331867729Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.333775_202.1","section":"bucket","name":"bbucnh-22","timestamp":"2021-03-12T02:36:41.333775173Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lRVZbkb3M5VDTFzWzb2Ao_n","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.338365_203.1","section":"bucket","name":"bbucnh-22","timestamp":"2021-03-12T02:36:41.338365599Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lRVZbkb3M5VDTFzWzb2Ao_n","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.375262_204.1","section":"bucket.instance","name":"bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32","timestamp":"2021-03-12T02:36:41.375262696Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.380184_205.1","section":"bucket.instance","name":"bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32","timestamp":"2021-03-12T02:36:41.380184001Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.382095_206.1","section":"bucket","name":"bbucnh-23","timestamp":"2021-03-12T02:36:41.382095207Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ox0KlvX6xNF4jdqzrljR_Yz","ver":1},"status":{"status":"write"}}},{"id":"1_1615516601.386448_207.1","section":"bucket","name":"bbucnh-23","timestamp":"2021-03-12T02:36:41.386448784Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ox0KlvX6xNF4jdqzrljR_Yz","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516601.406400_208.1","section":"bucket.instance","name":"bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30","timestamp":"2021-03-12T02:36:41.406400290Z","data":{"read_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":1},"write_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":2},"status":{"status":"write"}}},{"id":"1_1615516601.411130_209.1","section":"bucket.instance","name":"bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30","timestamp":"2021-03-12T02:36:41.411130339Z","data":{"read_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":1},"write_version":{"tag":"_Lk8gnpRVJ4X5scSP24k_ofl","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516601.415296_210.1","section":"bucket.instance","name":"bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31","timestamp":"2021-03-12T02:36:41.415296943Z","data":{"read_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":1},"write_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":2},"status":{"status":"write"}}},{"id":"1_1615516601.420232_211.1","section":"bucket.instance","name":"bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31","timestamp":"2021-03-12T02:36:41.420232785Z","data":{"read_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":1},"write_version":{"tag":"_VVLWBYKeNOSb4LRyv68T-Fi","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516601.429424_212.1","section":"bucket.instance","name":"bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32","timestamp":"2021-03-12T02:36:41.429424011Z","data":{"read_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":1},"write_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":2},"status":{"status":"write"}}},{"id":"1_1615516601.435061_213.1","section":"bucket.instance","name":"bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32","timestamp":"2021-03-12T02:36:41.435061595Z","data":{"read_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":1},"write_version":{"tag":"_aM5sOo2-cMn9mAc4eba0dIm","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516623.905134_226.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:03.905134670Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":1},"status":{"status":"write"}}},{"id":"1_1615516623.909582_227.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:03.909582485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516623.911054_228.1","section":"bucket","name":"bbucnh-24","timestamp":"2021-03-12T02:37:03.911054290Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kAh_Dud7mDE7666DS44bDVH","ver":1},"status":{"status":"write"}}},{"id":"1_1615516623.915168_229.1","section":"bucket","name":"bbucnh-24","timestamp":"2021-03-12T02:37:03.915168538Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kAh_Dud7mDE7666DS44bDVH","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516623.938779_230.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:03.938779082Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":1},"status":{"status":"write"}}},{"id":"1_1615516623.944057_231.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:03.944057002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516623.945869_232.1","section":"bucket","name":"bbucnh-25","timestamp":"2021-03-12T02:37:03.945869854Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FVWFGEedKW_N26l2XXHS__P","ver":1},"status":{"status":"write"}}},{"id":"1_1615516623.952222_233.1","section":"bucket","name":"bbucnh-25","timestamp":"2021-03-12T02:37:03.952222510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FVWFGEedKW_N26l2XXHS__P","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516623.998369_234.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:03.998369746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":1},"status":{"status":"write"}}},{"id":"1_1615516624.003251_235.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.003251527Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516624.004804_236.1","section":"bucket","name":"bbucnh-26","timestamp":"2021-03-12T02:37:04.004804195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pZWbNId4gDq01c4QPLoSkT3","ver":1},"status":{"status":"write"}}},{"id":"1_1615516624.009867_237.1","section":"bucket","name":"bbucnh-26","timestamp":"2021-03-12T02:37:04.009867850Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pZWbNId4gDq01c4QPLoSkT3","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516624.034129_238.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:04.034129873Z","data":{"read_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":1},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":2},"status":{"status":"write"}}},{"id":"1_1615516624.040808_239.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:04.040808023Z","data":{"read_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":1},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516624.044134_240.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:04.044134611Z","data":{"read_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":2},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":3},"status":{"status":"write"}}},{"id":"1_1615516624.051708_241.1","section":"bucket.instance","name":"bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33","timestamp":"2021-03-12T02:37:04.051708030Z","data":{"read_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":2},"write_version":{"tag":"_ApFjSxH2JjD-4vPdSFtySDO","ver":3},"status":{"status":"complete"}}},{"id":"1_1615516624.056887_242.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:04.056887996Z","data":{"read_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":1},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":2},"status":{"status":"write"}}},{"id":"1_1615516624.063220_243.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:04.063220593Z","data":{"read_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":1},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516624.073512_244.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:04.073512038Z","data":{"read_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":2},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":3},"status":{"status":"write"}}},{"id":"1_1615516624.080300_245.1","section":"bucket.instance","name":"bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34","timestamp":"2021-03-12T02:37:04.080300997Z","data":{"read_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":2},"write_version":{"tag":"_PUWYrd3xFGOumgicBFViMHT","ver":3},"status":{"status":"complete"}}},{"id":"1_1615516624.092046_246.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.092046934Z","data":{"read_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":1},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":2},"status":{"status":"write"}}},{"id":"1_1615516624.098396_247.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.098396048Z","data":{"read_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":1},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516624.109918_248.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.109918336Z","data":{"read_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":2},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":3},"status":{"status":"write"}}},{"id":"1_1615516624.119886_249.1","section":"bucket.instance","name":"bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35","timestamp":"2021-03-12T02:37:04.119886963Z","data":{"read_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":2},"write_version":{"tag":"_CkT15SQf5VdymBYVN4AQH0W","ver":3},"status":{"status":"complete"}}},{"id":"1_1615516648.953955_260.1","section":"bucket.instance","name":"bbucnh-27:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.36","timestamp":"2021-03-12T02:37:28.953955696Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516648.958378_261.1","section":"bucket.instance","name":"bbucnh-27:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.36","timestamp":"2021-03-12T02:37:28.958378293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516648.959965_262.1","section":"bucket","name":"bbucnh-27","timestamp":"2021-03-12T02:37:28.959965688Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FmfQYONhZDXgO1uw7EYs9_S","ver":1},"status":{"status":"write"}}},{"id":"1_1615516648.963902_263.1","section":"bucket","name":"bbucnh-27","timestamp":"2021-03-12T02:37:28.963902711Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FmfQYONhZDXgO1uw7EYs9_S","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516648.987286_264.1","section":"bucket.instance","name":"bbucnh-28:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37","timestamp":"2021-03-12T02:37:28.987286166Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":1},"status":{"status":"write"}}},{"id":"1_1615516648.993373_265.1","section":"bucket.instance","name":"bbucnh-28:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37","timestamp":"2021-03-12T02:37:28.993373542Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516648.995310_266.1","section":"bucket","name":"bbucnh-28","timestamp":"2021-03-12T02:37:28.995310434Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2gVXe3dHFNHJj_5mBzi7BFQ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.000050_267.1","section":"bucket","name":"bbucnh-28","timestamp":"2021-03-12T02:37:29.000050799Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2gVXe3dHFNHJj_5mBzi7BFQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.040216_268.1","section":"bucket.instance","name":"bbucnh-29:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38","timestamp":"2021-03-12T02:37:29.040216016Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.045685_269.1","section":"bucket.instance","name":"bbucnh-29:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38","timestamp":"2021-03-12T02:37:29.045685128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.047750_270.1","section":"bucket","name":"bbucnh-29","timestamp":"2021-03-12T02:37:29.047750134Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NzPM--eEDeYlpvhbh0MdeK4","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.052200_271.1","section":"bucket","name":"bbucnh-29","timestamp":"2021-03-12T02:37:29.052200209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NzPM--eEDeYlpvhbh0MdeK4","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.075303_272.1","section":"bucket.instance","name":"bbucnh-27:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.36","timestamp":"2021-03-12T02:37:29.075303587Z","data":{"read_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":1},"write_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.080878_273.1","section":"bucket.instance","name":"bbucnh-27:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.36","timestamp":"2021-03-12T02:37:29.080878607Z","data":{"read_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":1},"write_version":{"tag":"_XkoRxfJTKwwalsHF93poDkM","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.129020_274.1","section":"bucket.instance","name":"bbucnh-28:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37","timestamp":"2021-03-12T02:37:29.129020143Z","data":{"read_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":1},"write_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.137207_275.1","section":"bucket.instance","name":"bbucnh-28:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37","timestamp":"2021-03-12T02:37:29.137207384Z","data":{"read_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":1},"write_version":{"tag":"_GvLyvewDcujxwIBkjonj0UW","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.196723_276.1","section":"bucket.instance","name":"bbucnh-29:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38","timestamp":"2021-03-12T02:37:29.196723737Z","data":{"read_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":1},"write_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.204941_277.1","section":"bucket.instance","name":"bbucnh-29:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38","timestamp":"2021-03-12T02:37:29.204941290Z","data":{"read_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":1},"write_version":{"tag":"_iYAR-qFPjNm25A_huHG4BSp","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.273026_278.1","section":"bucket.instance","name":"bbucnh-30:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.39","timestamp":"2021-03-12T02:37:29.273026550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.278145_279.1","section":"bucket.instance","name":"bbucnh-30:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.39","timestamp":"2021-03-12T02:37:29.278145289Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.279975_280.1","section":"bucket","name":"bbucnh-30","timestamp":"2021-03-12T02:37:29.279975374Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ktkorz1HBIgiOiSGUck37Fi","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.284871_281.1","section":"bucket","name":"bbucnh-30","timestamp":"2021-03-12T02:37:29.284871749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ktkorz1HBIgiOiSGUck37Fi","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.312142_282.1","section":"bucket.instance","name":"bbucnh-31:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40","timestamp":"2021-03-12T02:37:29.312142460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.317109_283.1","section":"bucket.instance","name":"bbucnh-31:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40","timestamp":"2021-03-12T02:37:29.317109530Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.318899_284.1","section":"bucket","name":"bbucnh-31","timestamp":"2021-03-12T02:37:29.318899628Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RVIUeIv4NvsRyQ3Z9VfRls-","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.323537_285.1","section":"bucket","name":"bbucnh-31","timestamp":"2021-03-12T02:37:29.323537422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RVIUeIv4NvsRyQ3Z9VfRls-","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.364204_286.1","section":"bucket.instance","name":"bbucnh-32:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41","timestamp":"2021-03-12T02:37:29.364204861Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.369006_287.1","section":"bucket.instance","name":"bbucnh-32:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41","timestamp":"2021-03-12T02:37:29.369006270Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.370821_288.1","section":"bucket","name":"bbucnh-32","timestamp":"2021-03-12T02:37:29.370821226Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uNNdkr5c4kYGRpyB0KL4L6X","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.374864_289.1","section":"bucket","name":"bbucnh-32","timestamp":"2021-03-12T02:37:29.374864824Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uNNdkr5c4kYGRpyB0KL4L6X","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.400473_291.1","section":"bucket.instance","name":"bbucnh-30:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.39","timestamp":"2021-03-12T02:37:29.400473777Z","data":{"read_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":1},"write_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.405529_292.1","section":"bucket.instance","name":"bbucnh-30:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.39","timestamp":"2021-03-12T02:37:29.405529038Z","data":{"read_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":1},"write_version":{"tag":"_yWD0daH5T7M1Kq1al0hHmb4","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.454396_293.1","section":"bucket.instance","name":"bbucnh-31:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40","timestamp":"2021-03-12T02:37:29.454396735Z","data":{"read_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":1},"write_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.462159_294.1","section":"bucket.instance","name":"bbucnh-31:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40","timestamp":"2021-03-12T02:37:29.462159544Z","data":{"read_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":1},"write_version":{"tag":"_AuVEYVENWUs8ixpc90HM29g","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.526776_295.1","section":"bucket.instance","name":"bbucnh-32:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41","timestamp":"2021-03-12T02:37:29.526776297Z","data":{"read_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":1},"write_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.534441_296.1","section":"bucket.instance","name":"bbucnh-32:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41","timestamp":"2021-03-12T02:37:29.534441885Z","data":{"read_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":1},"write_version":{"tag":"_ZimM8bUVFszraxDRsZh-I2D","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516649.609051_298.1","section":"bucket.instance","name":"bbucnh-33:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.42","timestamp":"2021-03-12T02:37:29.609051947Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.613737_299.1","section":"bucket.instance","name":"bbucnh-33:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.42","timestamp":"2021-03-12T02:37:29.613737442Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.615385_300.1","section":"bucket","name":"bbucnh-33","timestamp":"2021-03-12T02:37:29.615385271Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_15Kd9p4ie5l3fyoCESSGjmB","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.619356_301.1","section":"bucket","name":"bbucnh-33","timestamp":"2021-03-12T02:37:29.619356209Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_15Kd9p4ie5l3fyoCESSGjmB","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.644293_302.1","section":"bucket.instance","name":"bbucnh-34:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43","timestamp":"2021-03-12T02:37:29.644293956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.649652_303.1","section":"bucket.instance","name":"bbucnh-34:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43","timestamp":"2021-03-12T02:37:29.649652088Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.651633_304.1","section":"bucket","name":"bbucnh-34","timestamp":"2021-03-12T02:37:29.651633615Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KxY1ghborztF7ijmoRBmfYR","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.656854_305.1","section":"bucket","name":"bbucnh-34","timestamp":"2021-03-12T02:37:29.656854451Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KxY1ghborztF7ijmoRBmfYR","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.700347_306.1","section":"bucket.instance","name":"bbucnh-35:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44","timestamp":"2021-03-12T02:37:29.700347440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.705720_307.1","section":"bucket.instance","name":"bbucnh-35:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44","timestamp":"2021-03-12T02:37:29.705720561Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.707370_308.1","section":"bucket","name":"bbucnh-35","timestamp":"2021-03-12T02:37:29.707370179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tJebU3psH_SbzeVFZfWinUh","ver":1},"status":{"status":"write"}}},{"id":"1_1615516649.711408_309.1","section":"bucket","name":"bbucnh-35","timestamp":"2021-03-12T02:37:29.711408051Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tJebU3psH_SbzeVFZfWinUh","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516649.734891_310.1","section":"bucket.instance","name":"bbucnh-33:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.42","timestamp":"2021-03-12T02:37:29.734891456Z","data":{"read_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":1},"write_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":2},"status":{"status":"write"}}},{"id":"1_1615516649.739920_311.1","section":"bucket.instance","name":"bbucnh-33:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.42","timestamp":"2021-03-12T02:37:29.739920185Z","data":{"read_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":1},"write_version":{"tag":"_7w6iGITmggZGJ0rfdwQZpwl","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516650.789737_312.1","section":"bucket.instance","name":"bbucnh-34:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43","timestamp":"2021-03-12T02:37:30.789737190Z","data":{"read_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":1},"write_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":2},"status":{"status":"write"}}},{"id":"1_1615516650.798206_313.1","section":"bucket.instance","name":"bbucnh-34:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43","timestamp":"2021-03-12T02:37:30.798206828Z","data":{"read_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":1},"write_version":{"tag":"_MphwsnXXgRCzaupENiRjbLM","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516652.335363_314.1","section":"bucket.instance","name":"bbucnh-35:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44","timestamp":"2021-03-12T02:37:32.335363586Z","data":{"read_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":1},"write_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":2},"status":{"status":"write"}}},{"id":"1_1615516652.344811_315.1","section":"bucket.instance","name":"bbucnh-35:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44","timestamp":"2021-03-12T02:37:32.344811119Z","data":{"read_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":1},"write_version":{"tag":"_kijRxbWcUArzD2GqDRFlSJk","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516653.069940_316.1","section":"bucket.instance","name":"bbucnh-36:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45","timestamp":"2021-03-12T02:37:33.069940667Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VuDCjriGUmI_jGsCkSwyMPM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516653.075083_317.1","section":"bucket.instance","name":"bbucnh-36:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45","timestamp":"2021-03-12T02:37:33.075083097Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VuDCjriGUmI_jGsCkSwyMPM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516653.076710_318.1","section":"bucket","name":"bbucnh-36","timestamp":"2021-03-12T02:37:33.076710866Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6lGSmQvlrfayDP125PAmHZF","ver":1},"status":{"status":"write"}}},{"id":"1_1615516653.081538_319.1","section":"bucket","name":"bbucnh-36","timestamp":"2021-03-12T02:37:33.081538100Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6lGSmQvlrfayDP125PAmHZF","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516653.664443_320.1","section":"bucket.instance","name":"bbucnh-37:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46","timestamp":"2021-03-12T02:37:33.664443824Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HmfgDDWFN0bOGRAMO2ByNk7","ver":1},"status":{"status":"write"}}},{"id":"1_1615516653.669629_321.1","section":"bucket.instance","name":"bbucnh-37:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46","timestamp":"2021-03-12T02:37:33.669629721Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HmfgDDWFN0bOGRAMO2ByNk7","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516653.671304_322.1","section":"bucket","name":"bbucnh-37","timestamp":"2021-03-12T02:37:33.671304239Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J6fc8NV-QkiOt2H3vUtcdSS","ver":1},"status":{"status":"write"}}},{"id":"1_1615516653.675804_323.1","section":"bucket","name":"bbucnh-37","timestamp":"2021-03-12T02:37:33.675804856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_J6fc8NV-QkiOt2H3vUtcdSS","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516653.998533_324.1","section":"bucket.instance","name":"bbucnh-38:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47","timestamp":"2021-03-12T02:37:33.998533211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DXFrPBq7cR1YlfkiP9rTC_h","ver":1},"status":{"status":"write"}}},{"id":"1_1615516654.002986_325.1","section":"bucket.instance","name":"bbucnh-38:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47","timestamp":"2021-03-12T02:37:34.002986261Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DXFrPBq7cR1YlfkiP9rTC_h","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516654.004526_326.1","section":"bucket","name":"bbucnh-38","timestamp":"2021-03-12T02:37:34.004526175Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B2U5VzBNff39d-uvhTwPTVA","ver":1},"status":{"status":"write"}}},{"id":"1_1615516654.008644_327.1","section":"bucket","name":"bbucnh-38","timestamp":"2021-03-12T02:37:34.008644344Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_B2U5VzBNff39d-uvhTwPTVA","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.472251_328.1","section":"bucket.instance","name":"bbucnh-39:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.48","timestamp":"2021-03-12T02:37:36.472251903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxwXPsBw5MBOuLy2DL0RzSC","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.477307_329.1","section":"bucket.instance","name":"bbucnh-39:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.48","timestamp":"2021-03-12T02:37:36.477307238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VxwXPsBw5MBOuLy2DL0RzSC","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.479321_330.1","section":"bucket","name":"bbucnh-39","timestamp":"2021-03-12T02:37:36.479321851Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7RmOoO6yS0MAc_ELYlYq8yN","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.483505_331.1","section":"bucket","name":"bbucnh-39","timestamp":"2021-03-12T02:37:36.483505123Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7RmOoO6yS0MAc_ELYlYq8yN","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.511042_332.1","section":"bucket.instance","name":"bbucnh-40:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.49","timestamp":"2021-03-12T02:37:36.511042128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q1cAgCTgGfyj4SRitg3YHYj","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.516504_333.1","section":"bucket.instance","name":"bbucnh-40:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.49","timestamp":"2021-03-12T02:37:36.516504986Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q1cAgCTgGfyj4SRitg3YHYj","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.518471_334.1","section":"bucket","name":"bbucnh-40","timestamp":"2021-03-12T02:37:36.518471862Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vmcO4AQPJh3F7zPA8Q1MDcO","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.522955_335.1","section":"bucket","name":"bbucnh-40","timestamp":"2021-03-12T02:37:36.522955683Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vmcO4AQPJh3F7zPA8Q1MDcO","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.569936_336.1","section":"bucket.instance","name":"bbucnh-41:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.50","timestamp":"2021-03-12T02:37:36.569936921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V9oej-UfPhBNN29T2LbubWl","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.574506_337.1","section":"bucket.instance","name":"bbucnh-41:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.50","timestamp":"2021-03-12T02:37:36.574506107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V9oej-UfPhBNN29T2LbubWl","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516656.576240_338.1","section":"bucket","name":"bbucnh-41","timestamp":"2021-03-12T02:37:36.576240547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TmSKoQq5CYyZMuTTOWMF8jB","ver":1},"status":{"status":"write"}}},{"id":"1_1615516656.580426_339.1","section":"bucket","name":"bbucnh-41","timestamp":"2021-03-12T02:37:36.580426570Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TmSKoQq5CYyZMuTTOWMF8jB","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516277.246267_81.1","section":"bucket.instance","name":"bbucnh-1:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.1","timestamp":"2021-03-12T02:31:17.246267120Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qfxaS5zdrJis58Zg6Obg75g","ver":1},"status":{"status":"write"}}},{"id":"1_1615516277.255262_85.1","section":"bucket.instance","name":"bbucnh-1:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.1","timestamp":"2021-03-12T02:31:17.255262459Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qfxaS5zdrJis58Zg6Obg75g","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516277.256664_86.1","section":"bucket","name":"bbucnh-1","timestamp":"2021-03-12T02:31:17.256664887Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SJQ8Ox0nk_KuM-tokbc3CHJ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516277.260851_87.1","section":"bucket","name":"bbucnh-1","timestamp":"2021-03-12T02:31:17.260851736Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_SJQ8Ox0nk_KuM-tokbc3CHJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516277.279301_88.1","section":"bucket.instance","name":"bbucnh-2:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.2","timestamp":"2021-03-12T02:31:17.279301358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OdmYkD6xh5KguUnb-mdZ1aM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516277.284169_89.1","section":"bucket.instance","name":"bbucnh-2:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.2","timestamp":"2021-03-12T02:31:17.284169621Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OdmYkD6xh5KguUnb-mdZ1aM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516277.285818_90.1","section":"bucket","name":"bbucnh-2","timestamp":"2021-03-12T02:31:17.285818007Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U61zBQN8Y4-IaOqXqUfnFDK","ver":1},"status":{"status":"write"}}},{"id":"1_1615516277.289902_91.1","section":"bucket","name":"bbucnh-2","timestamp":"2021-03-12T02:31:17.289902101Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_U61zBQN8Y4-IaOqXqUfnFDK","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516280.582981_92.1","section":"bucket.instance","name":"bbucnh-3:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.3","timestamp":"2021-03-12T02:31:20.582981563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WmExOXe_DXwUuw8ZWwStmCG","ver":1},"status":{"status":"write"}}},{"id":"1_1615516280.587282_93.1","section":"bucket.instance","name":"bbucnh-3:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.3","timestamp":"2021-03-12T02:31:20.587282731Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WmExOXe_DXwUuw8ZWwStmCG","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516280.588897_94.1","section":"bucket","name":"bbucnh-3","timestamp":"2021-03-12T02:31:20.588897050Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vTlspttjgtqvLuUNPCk_X1Z","ver":1},"status":{"status":"write"}}},{"id":"1_1615516280.593132_95.1","section":"bucket","name":"bbucnh-3","timestamp":"2021-03-12T02:31:20.593132328Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vTlspttjgtqvLuUNPCk_X1Z","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516289.394488_96.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.4","timestamp":"2021-03-12T02:31:29.394488830Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P9XE_bWiNIw3vvd5uTEfxr3","ver":1},"status":{"status":"write"}}},{"id":"1_1615516289.399511_98.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.4","timestamp":"2021-03-12T02:31:29.399511189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P9XE_bWiNIw3vvd5uTEfxr3","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516289.400904_99.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:29.400904315Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HJA5lWRMWyY0IaqyBfFi1N4","ver":1},"status":{"status":"write"}}},{"id":"1_1615516289.404420_100.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:29.404420265Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HJA5lWRMWyY0IaqyBfFi1N4","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516289.903936_101.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.5","timestamp":"2021-03-12T02:31:29.903936280Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1ecwCnmWKxBMRrpYQmvXMe-","ver":1},"status":{"status":"write"}}},{"id":"1_1615516289.908750_102.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.5","timestamp":"2021-03-12T02:31:29.908750593Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1ecwCnmWKxBMRrpYQmvXMe-","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516289.910808_103.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:29.910808743Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nZiOpvQgZemMdcUiOXIn1VT","ver":1},"status":{"status":"write"}}},{"id":"1_1615516289.916752_104.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:29.916752691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nZiOpvQgZemMdcUiOXIn1VT","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516290.449434_105.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.6","timestamp":"2021-03-12T02:31:30.449434628Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rHxZdUR046Wgm3Zafw0Xi8l","ver":1},"status":{"status":"write"}}},{"id":"1_1615516290.453654_106.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.6","timestamp":"2021-03-12T02:31:30.453654797Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rHxZdUR046Wgm3Zafw0Xi8l","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516290.455139_107.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:30.455139379Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sg78H-Zgxy2C6QwruYgc-Rg","ver":1},"status":{"status":"write"}}},{"id":"1_1615516290.459112_108.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:30.459112128Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sg78H-Zgxy2C6QwruYgc-Rg","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.676123_109.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.7","timestamp":"2021-03-12T02:31:32.676123449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T4BoTd-m-PH42nJonMN7kZP","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.681307_111.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.7","timestamp":"2021-03-12T02:31:32.681307323Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T4BoTd-m-PH42nJonMN7kZP","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.683099_112.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:32.683099550Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EBuzwkBweLN5rvCeTKt4REO","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.688466_113.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:32.688466865Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EBuzwkBweLN5rvCeTKt4REO","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.712314_114.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.8","timestamp":"2021-03-12T02:31:32.712314715Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c8czZW2QuWkq0wJE4l-Nvwz","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.716801_115.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.8","timestamp":"2021-03-12T02:31:32.716801906Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c8czZW2QuWkq0wJE4l-Nvwz","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.718398_116.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:32.718398180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yjAubSLjTLPIuEzKEWdFJD8","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.721131_117.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:32.721131522Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yjAubSLjTLPIuEzKEWdFJD8","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.742288_118.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.9","timestamp":"2021-03-12T02:31:32.742288024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cd_ZmHPW-yz7g9wMfu8z1OM","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.746471_119.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.9","timestamp":"2021-03-12T02:31:32.746471019Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cd_ZmHPW-yz7g9wMfu8z1OM","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.748067_120.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:32.748067528Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yyCMzqqMnUhRq-wn-e0Oe_1","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.752347_121.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:32.752347222Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yyCMzqqMnUhRq-wn-e0Oe_1","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.782913_122.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.10","timestamp":"2021-03-12T02:31:32.782913622Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cbSranN-_o_cTrR-FcWF_2Y","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.788130_123.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.10","timestamp":"2021-03-12T02:31:32.788130810Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cbSranN-_o_cTrR-FcWF_2Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.790158_124.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:32.790158153Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7PSv7RO8pR2twgmTFBWqx0N","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.795541_125.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:32.795541882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7PSv7RO8pR2twgmTFBWqx0N","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.844603_126.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.11","timestamp":"2021-03-12T02:31:32.844603751Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4_B8b59PwgOUx0apcPz4YE3","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.849848_127.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.11","timestamp":"2021-03-12T02:31:32.849848924Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4_B8b59PwgOUx0apcPz4YE3","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.851664_128.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:32.851664510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G_5ocbuuJ8e81M9Oey3KkUa","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.855275_129.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:32.855275037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G_5ocbuuJ8e81M9Oey3KkUa","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516292.947746_130.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.12","timestamp":"2021-03-12T02:31:32.947746750Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hLAW_-1vXKVBgOh8Z3yp4RR","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.952705_131.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.12","timestamp":"2021-03-12T02:31:32.952705876Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hLAW_-1vXKVBgOh8Z3yp4RR","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516292.954374_132.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:32.954374195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u-RVInQ17b5MGi2bJAMXC9c","ver":1},"status":{"status":"write"}}},{"id":"1_1615516292.957220_133.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:32.957220699Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u-RVInQ17b5MGi2bJAMXC9c","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516293.053188_134.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.13","timestamp":"2021-03-12T02:31:33.053188210Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3Sm3iR_O9gOws1Qm0XHBesI","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.059103_135.1","section":"bucket.instance","name":"bbucnh-4:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.13","timestamp":"2021-03-12T02:31:33.059103540Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3Sm3iR_O9gOws1Qm0XHBesI","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516293.060699_136.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:33.060699635Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fJHHqFsf0ZnQ8HA3eVv0HsQ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.063543_137.1","section":"bucket","name":"bbucnh-4","timestamp":"2021-03-12T02:31:33.063543264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fJHHqFsf0ZnQ8HA3eVv0HsQ","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516293.100428_138.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.14","timestamp":"2021-03-12T02:31:33.100428411Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qEcq5zwRQGvPWI89_dceYVx","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.105635_139.1","section":"bucket.instance","name":"bbucnh-5:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.14","timestamp":"2021-03-12T02:31:33.105635408Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_qEcq5zwRQGvPWI89_dceYVx","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516293.107322_140.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:33.107322438Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_usXxiQpihhvHv9w9H2KqHQL","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.112787_141.1","section":"bucket","name":"bbucnh-5","timestamp":"2021-03-12T02:31:33.112787593Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_usXxiQpihhvHv9w9H2KqHQL","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516293.205051_142.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.15","timestamp":"2021-03-12T02:31:33.205051135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NyLEienNgXaYmXOscru4QK2","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.210392_143.1","section":"bucket.instance","name":"bbucnh-6:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.15","timestamp":"2021-03-12T02:31:33.210392899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NyLEienNgXaYmXOscru4QK2","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516293.212220_144.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:33.212220343Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_McGu-bUTB7Iwrzro3KX9U6v","ver":1},"status":{"status":"write"}}},{"id":"1_1615516293.216631_145.1","section":"bucket","name":"bbucnh-6","timestamp":"2021-03-12T02:31:33.216631468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_McGu-bUTB7Iwrzro3KX9U6v","ver":1},"status":{"status":"abort"}}},{"id":"1_1615516300.493202_146.1","section":"bucket.instance","name":"bbucnh-7:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.16","timestamp":"2021-03-12T02:31:40.493202575Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJvoXXdfbZSbk6J8UPJzIDB","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.498785_148.1","section":"bucket.instance","name":"bbucnh-7:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.16","timestamp":"2021-03-12T02:31:40.498785950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kJvoXXdfbZSbk6J8UPJzIDB","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.500553_149.1","section":"bucket","name":"bbucnh-7","timestamp":"2021-03-12T02:31:40.500553657Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.507000_150.1","section":"bucket","name":"bbucnh-7","timestamp":"2021-03-12T02:31:40.507000938Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.528275_152.1","section":"bucket.instance","name":"bbucnh-8:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.17","timestamp":"2021-03-12T02:31:40.528275074Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n22OShRJQ_FxFcZr65eyDbq","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.533734_153.1","section":"bucket.instance","name":"bbucnh-8:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.17","timestamp":"2021-03-12T02:31:40.533734931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n22OShRJQ_FxFcZr65eyDbq","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.535436_154.1","section":"bucket","name":"bbucnh-8","timestamp":"2021-03-12T02:31:40.535436833Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.541545_155.1","section":"bucket","name":"bbucnh-8","timestamp":"2021-03-12T02:31:40.541545383Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.583042_156.1","section":"bucket.instance","name":"bbucnh-9:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.18","timestamp":"2021-03-12T02:31:40.583042072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WFleci_lBAEwlmYvNA5F_z0","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.587676_157.1","section":"bucket.instance","name":"bbucnh-9:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.18","timestamp":"2021-03-12T02:31:40.587676557Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WFleci_lBAEwlmYvNA5F_z0","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516300.589578_158.1","section":"bucket","name":"bbucnh-9","timestamp":"2021-03-12T02:31:40.589578574Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":1},"status":{"status":"write"}}},{"id":"1_1615516300.595034_159.1","section":"bucket","name":"bbucnh-9","timestamp":"2021-03-12T02:31:40.595034275Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516309.326188_162.1","section":"bucket","name":"bbucnh-7","timestamp":"2021-03-12T02:31:49.326188604Z","data":{"read_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":1},"write_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":2},"status":{"status":"remove"}}},{"id":"1_1615516309.332337_163.1","section":"bucket","name":"bbucnh-7","timestamp":"2021-03-12T02:31:49.332337164Z","data":{"read_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":1},"write_version":{"tag":"_zI1LzPmvXCpEJa-yjcjeo8e","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516309.508827_165.1","section":"bucket","name":"bbucnh-8","timestamp":"2021-03-12T02:31:49.508827513Z","data":{"read_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":1},"write_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":2},"status":{"status":"remove"}}},{"id":"1_1615516309.514944_166.1","section":"bucket","name":"bbucnh-8","timestamp":"2021-03-12T02:31:49.514944882Z","data":{"read_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":1},"write_version":{"tag":"_fHfDDv-O4xoZY3Mp4bDaAzE","ver":2},"status":{"status":"complete"}}},{"id":"1_1615516309.592520_167.1","section":"bucket","name":"bbucnh-9","timestamp":"2021-03-12T02:31:49.592520057Z","data":{"read_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":1},"write_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":2},"status":{"status":"remove"}}},{"id":"1_1615516309.597935_168.1","section":"bucket","name":"bbucnh-9","timestamp":"2021-03-12T02:31:49.597935909Z","data":{"read_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":1},"write_version":{"tag":"_LZwbxsPUNyPRqocsuLeWwzI","ver":2},"status":{"status":"complete"}}}] 2021-03-12T02:40:10.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '76eba794-fdc5-4e5e-9ce6-5f5a18a97795', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:10.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '76eba794-fdc5-4e5e-9ce6-5f5a18a97795', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:10.997 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 76eba794-fdc5-4e5e-9ce6-5f5a18a97795 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:11.037 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:11.038 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:11.265 INFO:teuthology.orchestra.run.smithi077.stdout:[{"id":"1_1615516729.434727_25.1","section":"bucket.instance","name":"bbucnh-42:374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.1","timestamp":"2021-03-12T02:38:49.434727385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ORmIXP1zJir4NYXDnujSado","ver":1},"status":{"status":"write"}}},{"id":"1_1615516729.439540_26.1","section":"bucket.instance","name":"bbucnh-42:374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.1","timestamp":"2021-03-12T02:38:49.439540582Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ORmIXP1zJir4NYXDnujSado","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516729.440925_27.1","section":"bucket","name":"bbucnh-42","timestamp":"2021-03-12T02:38:49.440925407Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gVy9mU-KY0Lk51q5C58S70N","ver":1},"status":{"status":"write"}}},{"id":"1_1615516729.444531_28.1","section":"bucket","name":"bbucnh-42","timestamp":"2021-03-12T02:38:49.444531907Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gVy9mU-KY0Lk51q5C58S70N","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516729.483161_29.1","section":"bucket.instance","name":"bbucnh-43:374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.2","timestamp":"2021-03-12T02:38:49.483161500Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9p1TNd3GUTiYHv5_kzrabF2","ver":1},"status":{"status":"write"}}},{"id":"1_1615516729.487090_30.1","section":"bucket.instance","name":"bbucnh-43:374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.2","timestamp":"2021-03-12T02:38:49.487090348Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9p1TNd3GUTiYHv5_kzrabF2","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516729.488459_31.1","section":"bucket","name":"bbucnh-43","timestamp":"2021-03-12T02:38:49.488459950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6WqdK9490TPRkTLREKNhsWf","ver":1},"status":{"status":"write"}}},{"id":"1_1615516729.491953_32.1","section":"bucket","name":"bbucnh-43","timestamp":"2021-03-12T02:38:49.491953043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6WqdK9490TPRkTLREKNhsWf","ver":1},"status":{"status":"complete"}}}] 2021-03-12T02:40:11.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '76eba794-fdc5-4e5e-9ce6-5f5a18a97795', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:11.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '76eba794-fdc5-4e5e-9ce6-5f5a18a97795', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:11.267 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 76eba794-fdc5-4e5e-9ce6-5f5a18a97795 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:11.307 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:11.308 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:11.516 INFO:teuthology.orchestra.run.smithi114.stdout:[{"id":"1_1615516729.434727_25.1","section":"bucket.instance","name":"bbucnh-42:374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.1","timestamp":"2021-03-12T02:38:49.434727385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ORmIXP1zJir4NYXDnujSado","ver":1},"status":{"status":"write"}}},{"id":"1_1615516729.439540_26.1","section":"bucket.instance","name":"bbucnh-42:374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.1","timestamp":"2021-03-12T02:38:49.439540582Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ORmIXP1zJir4NYXDnujSado","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516729.440925_27.1","section":"bucket","name":"bbucnh-42","timestamp":"2021-03-12T02:38:49.440925407Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gVy9mU-KY0Lk51q5C58S70N","ver":1},"status":{"status":"write"}}},{"id":"1_1615516729.444531_28.1","section":"bucket","name":"bbucnh-42","timestamp":"2021-03-12T02:38:49.444531907Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gVy9mU-KY0Lk51q5C58S70N","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516729.483161_29.1","section":"bucket.instance","name":"bbucnh-43:374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.2","timestamp":"2021-03-12T02:38:49.483161500Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9p1TNd3GUTiYHv5_kzrabF2","ver":1},"status":{"status":"write"}}},{"id":"1_1615516729.487090_30.1","section":"bucket.instance","name":"bbucnh-43:374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.2","timestamp":"2021-03-12T02:38:49.487090348Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9p1TNd3GUTiYHv5_kzrabF2","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516729.488459_31.1","section":"bucket","name":"bbucnh-43","timestamp":"2021-03-12T02:38:49.488459950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6WqdK9490TPRkTLREKNhsWf","ver":1},"status":{"status":"write"}}},{"id":"1_1615516729.491953_32.1","section":"bucket","name":"bbucnh-43","timestamp":"2021-03-12T02:38:49.491953043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6WqdK9490TPRkTLREKNhsWf","ver":1},"status":{"status":"complete"}}}] 2021-03-12T02:40:11.517 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '76eba794-fdc5-4e5e-9ce6-5f5a18a97795', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:11.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '76eba794-fdc5-4e5e-9ce6-5f5a18a97795', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:11.518 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 76eba794-fdc5-4e5e-9ce6-5f5a18a97795 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:11.558 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:11.558 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:11.785 INFO:teuthology.orchestra.run.smithi077.stdout:[{"id":"1_1615516729.434727_25.1","section":"bucket.instance","name":"bbucnh-42:374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.1","timestamp":"2021-03-12T02:38:49.434727385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ORmIXP1zJir4NYXDnujSado","ver":1},"status":{"status":"write"}}},{"id":"1_1615516729.439540_26.1","section":"bucket.instance","name":"bbucnh-42:374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.1","timestamp":"2021-03-12T02:38:49.439540582Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ORmIXP1zJir4NYXDnujSado","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516729.440925_27.1","section":"bucket","name":"bbucnh-42","timestamp":"2021-03-12T02:38:49.440925407Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gVy9mU-KY0Lk51q5C58S70N","ver":1},"status":{"status":"write"}}},{"id":"1_1615516729.444531_28.1","section":"bucket","name":"bbucnh-42","timestamp":"2021-03-12T02:38:49.444531907Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gVy9mU-KY0Lk51q5C58S70N","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516729.483161_29.1","section":"bucket.instance","name":"bbucnh-43:374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.2","timestamp":"2021-03-12T02:38:49.483161500Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9p1TNd3GUTiYHv5_kzrabF2","ver":1},"status":{"status":"write"}}},{"id":"1_1615516729.487090_30.1","section":"bucket.instance","name":"bbucnh-43:374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.2","timestamp":"2021-03-12T02:38:49.487090348Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9p1TNd3GUTiYHv5_kzrabF2","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516729.488459_31.1","section":"bucket","name":"bbucnh-43","timestamp":"2021-03-12T02:38:49.488459950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6WqdK9490TPRkTLREKNhsWf","ver":1},"status":{"status":"write"}}},{"id":"1_1615516729.491953_32.1","section":"bucket","name":"bbucnh-43","timestamp":"2021-03-12T02:38:49.491953043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6WqdK9490TPRkTLREKNhsWf","ver":1},"status":{"status":"complete"}}}] 2021-03-12T02:40:11.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '76eba794-fdc5-4e5e-9ce6-5f5a18a97795', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:11.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '76eba794-fdc5-4e5e-9ce6-5f5a18a97795', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:11.787 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 76eba794-fdc5-4e5e-9ce6-5f5a18a97795 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:11.828 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:12.036 INFO:teuthology.orchestra.run.smithi114.stdout:[{"id":"1_1615516729.434727_25.1","section":"bucket.instance","name":"bbucnh-42:374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.1","timestamp":"2021-03-12T02:38:49.434727385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ORmIXP1zJir4NYXDnujSado","ver":1},"status":{"status":"write"}}},{"id":"1_1615516729.439540_26.1","section":"bucket.instance","name":"bbucnh-42:374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.1","timestamp":"2021-03-12T02:38:49.439540582Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ORmIXP1zJir4NYXDnujSado","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516729.440925_27.1","section":"bucket","name":"bbucnh-42","timestamp":"2021-03-12T02:38:49.440925407Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gVy9mU-KY0Lk51q5C58S70N","ver":1},"status":{"status":"write"}}},{"id":"1_1615516729.444531_28.1","section":"bucket","name":"bbucnh-42","timestamp":"2021-03-12T02:38:49.444531907Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gVy9mU-KY0Lk51q5C58S70N","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516729.483161_29.1","section":"bucket.instance","name":"bbucnh-43:374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.2","timestamp":"2021-03-12T02:38:49.483161500Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9p1TNd3GUTiYHv5_kzrabF2","ver":1},"status":{"status":"write"}}},{"id":"1_1615516729.487090_30.1","section":"bucket.instance","name":"bbucnh-43:374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.2","timestamp":"2021-03-12T02:38:49.487090348Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9p1TNd3GUTiYHv5_kzrabF2","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516729.488459_31.1","section":"bucket","name":"bbucnh-43","timestamp":"2021-03-12T02:38:49.488459950Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6WqdK9490TPRkTLREKNhsWf","ver":1},"status":{"status":"write"}}},{"id":"1_1615516729.491953_32.1","section":"bucket","name":"bbucnh-43","timestamp":"2021-03-12T02:38:49.491953043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6WqdK9490TPRkTLREKNhsWf","ver":1},"status":{"status":"complete"}}}] 2021-03-12T02:40:12.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '8d2c0f1d-7035-4313-b22b-345f1f4f81db', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:12.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '8d2c0f1d-7035-4313-b22b-345f1f4f81db', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:12.037 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 8d2c0f1d-7035-4313-b22b-345f1f4f81db --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:12.077 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:12.078 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:12.295 INFO:teuthology.orchestra.run.smithi077.stdout:[{"id":"1_1615516795.720074_557.1","section":"bucket.instance","name":"bbucnh-44:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.1","timestamp":"2021-03-12T02:39:55.720074557Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dQqjTvKaXMOy6ANOmc9vWMJ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516795.724701_558.1","section":"bucket.instance","name":"bbucnh-44:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.1","timestamp":"2021-03-12T02:39:55.724701790Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dQqjTvKaXMOy6ANOmc9vWMJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516795.726380_559.1","section":"bucket","name":"bbucnh-44","timestamp":"2021-03-12T02:39:55.726380930Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x3nbKw_Cs3ULSmLdEjXx_Mw","ver":1},"status":{"status":"write"}}},{"id":"1_1615516795.730819_560.1","section":"bucket","name":"bbucnh-44","timestamp":"2021-03-12T02:39:55.730819870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x3nbKw_Cs3ULSmLdEjXx_Mw","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516795.768814_561.1","section":"bucket.instance","name":"bbucnh-45:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.2","timestamp":"2021-03-12T02:39:55.768814855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1xaoIXSmrFYIZYXneqo2lil","ver":1},"status":{"status":"write"}}},{"id":"1_1615516795.774253_562.1","section":"bucket.instance","name":"bbucnh-45:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.2","timestamp":"2021-03-12T02:39:55.774253668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1xaoIXSmrFYIZYXneqo2lil","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516795.776196_563.1","section":"bucket","name":"bbucnh-45","timestamp":"2021-03-12T02:39:55.776196394Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Exvi78Pm_fVRE4-8fCD8Lrc","ver":1},"status":{"status":"write"}}},{"id":"1_1615516795.780925_564.1","section":"bucket","name":"bbucnh-45","timestamp":"2021-03-12T02:39:55.780925617Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Exvi78Pm_fVRE4-8fCD8Lrc","ver":1},"status":{"status":"complete"}}}] 2021-03-12T02:40:12.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '8d2c0f1d-7035-4313-b22b-345f1f4f81db', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:12.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '8d2c0f1d-7035-4313-b22b-345f1f4f81db', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:12.297 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 8d2c0f1d-7035-4313-b22b-345f1f4f81db --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:12.336 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:12.336 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:12.541 INFO:teuthology.orchestra.run.smithi114.stdout:[{"id":"1_1615516795.720074_557.1","section":"bucket.instance","name":"bbucnh-44:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.1","timestamp":"2021-03-12T02:39:55.720074557Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dQqjTvKaXMOy6ANOmc9vWMJ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516795.724701_558.1","section":"bucket.instance","name":"bbucnh-44:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.1","timestamp":"2021-03-12T02:39:55.724701790Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dQqjTvKaXMOy6ANOmc9vWMJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516795.726380_559.1","section":"bucket","name":"bbucnh-44","timestamp":"2021-03-12T02:39:55.726380930Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x3nbKw_Cs3ULSmLdEjXx_Mw","ver":1},"status":{"status":"write"}}},{"id":"1_1615516795.730819_560.1","section":"bucket","name":"bbucnh-44","timestamp":"2021-03-12T02:39:55.730819870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x3nbKw_Cs3ULSmLdEjXx_Mw","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516795.768814_561.1","section":"bucket.instance","name":"bbucnh-45:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.2","timestamp":"2021-03-12T02:39:55.768814855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1xaoIXSmrFYIZYXneqo2lil","ver":1},"status":{"status":"write"}}},{"id":"1_1615516795.774253_562.1","section":"bucket.instance","name":"bbucnh-45:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.2","timestamp":"2021-03-12T02:39:55.774253668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1xaoIXSmrFYIZYXneqo2lil","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516795.776196_563.1","section":"bucket","name":"bbucnh-45","timestamp":"2021-03-12T02:39:55.776196394Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Exvi78Pm_fVRE4-8fCD8Lrc","ver":1},"status":{"status":"write"}}},{"id":"1_1615516795.780925_564.1","section":"bucket","name":"bbucnh-45","timestamp":"2021-03-12T02:39:55.780925617Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Exvi78Pm_fVRE4-8fCD8Lrc","ver":1},"status":{"status":"complete"}}}] 2021-03-12T02:40:12.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '8d2c0f1d-7035-4313-b22b-345f1f4f81db', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:12.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '8d2c0f1d-7035-4313-b22b-345f1f4f81db', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:12.543 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 8d2c0f1d-7035-4313-b22b-345f1f4f81db --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:12.583 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:12.584 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:12.813 INFO:teuthology.orchestra.run.smithi077.stdout:[{"id":"1_1615516795.720074_557.1","section":"bucket.instance","name":"bbucnh-44:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.1","timestamp":"2021-03-12T02:39:55.720074557Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dQqjTvKaXMOy6ANOmc9vWMJ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516795.724701_558.1","section":"bucket.instance","name":"bbucnh-44:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.1","timestamp":"2021-03-12T02:39:55.724701790Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dQqjTvKaXMOy6ANOmc9vWMJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516795.726380_559.1","section":"bucket","name":"bbucnh-44","timestamp":"2021-03-12T02:39:55.726380930Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x3nbKw_Cs3ULSmLdEjXx_Mw","ver":1},"status":{"status":"write"}}},{"id":"1_1615516795.730819_560.1","section":"bucket","name":"bbucnh-44","timestamp":"2021-03-12T02:39:55.730819870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x3nbKw_Cs3ULSmLdEjXx_Mw","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516795.768814_561.1","section":"bucket.instance","name":"bbucnh-45:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.2","timestamp":"2021-03-12T02:39:55.768814855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1xaoIXSmrFYIZYXneqo2lil","ver":1},"status":{"status":"write"}}},{"id":"1_1615516795.774253_562.1","section":"bucket.instance","name":"bbucnh-45:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.2","timestamp":"2021-03-12T02:39:55.774253668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1xaoIXSmrFYIZYXneqo2lil","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516795.776196_563.1","section":"bucket","name":"bbucnh-45","timestamp":"2021-03-12T02:39:55.776196394Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Exvi78Pm_fVRE4-8fCD8Lrc","ver":1},"status":{"status":"write"}}},{"id":"1_1615516795.780925_564.1","section":"bucket","name":"bbucnh-45","timestamp":"2021-03-12T02:39:55.780925617Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Exvi78Pm_fVRE4-8fCD8Lrc","ver":1},"status":{"status":"complete"}}}] 2021-03-12T02:40:12.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '8d2c0f1d-7035-4313-b22b-345f1f4f81db', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:12.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '8d2c0f1d-7035-4313-b22b-345f1f4f81db', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:12.815 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 8d2c0f1d-7035-4313-b22b-345f1f4f81db --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:12.852 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:12.852 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:13.063 INFO:teuthology.orchestra.run.smithi114.stdout:[{"id":"1_1615516795.720074_557.1","section":"bucket.instance","name":"bbucnh-44:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.1","timestamp":"2021-03-12T02:39:55.720074557Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dQqjTvKaXMOy6ANOmc9vWMJ","ver":1},"status":{"status":"write"}}},{"id":"1_1615516795.724701_558.1","section":"bucket.instance","name":"bbucnh-44:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.1","timestamp":"2021-03-12T02:39:55.724701790Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dQqjTvKaXMOy6ANOmc9vWMJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516795.726380_559.1","section":"bucket","name":"bbucnh-44","timestamp":"2021-03-12T02:39:55.726380930Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x3nbKw_Cs3ULSmLdEjXx_Mw","ver":1},"status":{"status":"write"}}},{"id":"1_1615516795.730819_560.1","section":"bucket","name":"bbucnh-44","timestamp":"2021-03-12T02:39:55.730819870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x3nbKw_Cs3ULSmLdEjXx_Mw","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516795.768814_561.1","section":"bucket.instance","name":"bbucnh-45:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.2","timestamp":"2021-03-12T02:39:55.768814855Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1xaoIXSmrFYIZYXneqo2lil","ver":1},"status":{"status":"write"}}},{"id":"1_1615516795.774253_562.1","section":"bucket.instance","name":"bbucnh-45:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.2","timestamp":"2021-03-12T02:39:55.774253668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1xaoIXSmrFYIZYXneqo2lil","ver":1},"status":{"status":"complete"}}},{"id":"1_1615516795.776196_563.1","section":"bucket","name":"bbucnh-45","timestamp":"2021-03-12T02:39:55.776196394Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Exvi78Pm_fVRE4-8fCD8Lrc","ver":1},"status":{"status":"write"}}},{"id":"1_1615516795.780925_564.1","section":"bucket","name":"bbucnh-45","timestamp":"2021-03-12T02:39:55.780925617Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Exvi78Pm_fVRE4-8fCD8Lrc","ver":1},"status":{"status":"complete"}}}] 2021-03-12T02:40:13.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:13.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:13.065 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:40:13.100 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:13.101 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:13.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:13.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:13.352 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:40:13.386 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:13.387 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:13.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:13.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:13.726 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:40:13.763 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:13.763 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:13.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:13.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:13.990 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:40:14.025 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:14.025 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:14.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0cf3ee11-29e6-4776-941b-50bda62adf82', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:14.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0cf3ee11-29e6-4776-941b-50bda62adf82', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:14.331 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0cf3ee11-29e6-4776-941b-50bda62adf82 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:14.370 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:14.370 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:14.600 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2021-03-12T02:40:14.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0cf3ee11-29e6-4776-941b-50bda62adf82', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:14.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0cf3ee11-29e6-4776-941b-50bda62adf82', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:14.602 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0cf3ee11-29e6-4776-941b-50bda62adf82 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:14.640 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:14.640 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:14.861 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2021-03-12T02:40:14.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '0cf3ee11-29e6-4776-941b-50bda62adf82', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:14.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '0cf3ee11-29e6-4776-941b-50bda62adf82', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:14.863 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 0cf3ee11-29e6-4776-941b-50bda62adf82 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:14.900 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:14.901 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:15.115 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2021-03-12T02:40:15.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '0cf3ee11-29e6-4776-941b-50bda62adf82', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:15.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '0cf3ee11-29e6-4776-941b-50bda62adf82', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:15.116 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 0cf3ee11-29e6-4776-941b-50bda62adf82 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:15.155 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:15.367 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2021-03-12T02:40:15.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '76eba794-fdc5-4e5e-9ce6-5f5a18a97795', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:15.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '76eba794-fdc5-4e5e-9ce6-5f5a18a97795', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:15.369 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 76eba794-fdc5-4e5e-9ce6-5f5a18a97795 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:15.409 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:15.409 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:15.622 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2021-03-12T02:40:15.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '76eba794-fdc5-4e5e-9ce6-5f5a18a97795', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:15.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '76eba794-fdc5-4e5e-9ce6-5f5a18a97795', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:15.624 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 76eba794-fdc5-4e5e-9ce6-5f5a18a97795 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:15.664 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:15.664 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:15.869 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2021-03-12T02:40:15.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '76eba794-fdc5-4e5e-9ce6-5f5a18a97795', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:15.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '76eba794-fdc5-4e5e-9ce6-5f5a18a97795', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:15.870 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 76eba794-fdc5-4e5e-9ce6-5f5a18a97795 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:15.913 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:15.913 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:16.111 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2021-03-12T02:40:16.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '76eba794-fdc5-4e5e-9ce6-5f5a18a97795', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:16.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '76eba794-fdc5-4e5e-9ce6-5f5a18a97795', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:16.114 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 76eba794-fdc5-4e5e-9ce6-5f5a18a97795 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:16.156 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:16.156 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:16.361 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2021-03-12T02:40:16.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '8d2c0f1d-7035-4313-b22b-345f1f4f81db', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:16.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '8d2c0f1d-7035-4313-b22b-345f1f4f81db', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:16.362 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 8d2c0f1d-7035-4313-b22b-345f1f4f81db --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:16.400 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:16.401 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:16.620 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2021-03-12T02:40:16.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '8d2c0f1d-7035-4313-b22b-345f1f4f81db', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:16.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '8d2c0f1d-7035-4313-b22b-345f1f4f81db', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:16.622 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 8d2c0f1d-7035-4313-b22b-345f1f4f81db --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:16.664 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:16.664 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:16.885 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2021-03-12T02:40:16.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '8d2c0f1d-7035-4313-b22b-345f1f4f81db', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:16.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '8d2c0f1d-7035-4313-b22b-345f1f4f81db', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:16.887 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 8d2c0f1d-7035-4313-b22b-345f1f4f81db --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:16.926 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:16.926 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:17.141 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2021-03-12T02:40:17.142 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '8d2c0f1d-7035-4313-b22b-345f1f4f81db', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:17.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '8d2c0f1d-7035-4313-b22b-345f1f4f81db', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:17.143 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 8d2c0f1d-7035-4313-b22b-345f1f4f81db --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:17.182 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:17.183 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:17.382 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2021-03-12T02:40:17.383 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2021-03-12T02:40:17.384 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-46 2021-03-12T02:40:17.772 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-47 2021-03-12T02:40:18.694 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-48 2021-03-12T02:40:19.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:19.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:19.340 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:19.379 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:19.606 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516456.957047_291.3","last_update":"2021-03-12T02:34:16.957047Z"},{"marker":"1_1615516819.208546_425.1","last_update":"2021-03-12T02:40:19.208546Z"},{"marker":"1_1615516500.963444_203.1","last_update":"2021-03-12T02:35:00.963444Z"},{"marker":"1_1615516657.578248_287.1","last_update":"2021-03-12T02:37:37.578248Z"}] 2021-03-12T02:40:19.607 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-12T02:40:19.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:19.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:19.608 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:19.651 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:19.651 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:19.862 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8376578737807985417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516654.983743_327.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:34.983743Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615516500.963444_203.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:00.963444Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.578248_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.578248Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:40:19.862 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=1 master=1_1615516819.208546_425.1 target=1_1615516654.983743_327.2 2021-03-12T02:40:24.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:24.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:24.866 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:24.907 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:24.908 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:25.108 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8376578737807985417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.617831_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.617831Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615516500.963444_203.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:00.963444Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.578248_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.578248Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:40:25.109 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-12T02:40:25.110 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:25.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:25.111 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:25.151 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:25.358 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516456.957047_291.3","last_update":"2021-03-12T02:34:16.957047Z"},{"marker":"1_1615516820.617831_435.1","last_update":"2021-03-12T02:40:20.617831Z"},{"marker":"1_1615516500.963444_203.1","last_update":"2021-03-12T02:35:00.963444Z"},{"marker":"1_1615516657.578248_287.1","last_update":"2021-03-12T02:37:37.578248Z"}] 2021-03-12T02:40:25.358 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-12T02:40:25.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:25.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:25.360 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:25.442 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:25.442 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:25.655 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1279455150295362088},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.617831_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.617831Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615516500.963444_203.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:00.963444Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.578248_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.578248Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:40:25.656 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-12T02:40:25.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:25.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:25.661 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:25.733 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:25.733 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:25.945 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516456.957047_291.3","last_update":"2021-03-12T02:34:16.957047Z"},{"marker":"1_1615516820.617831_435.1","last_update":"2021-03-12T02:40:20.617831Z"},{"marker":"1_1615516500.963444_203.1","last_update":"2021-03-12T02:35:00.963444Z"},{"marker":"1_1615516657.578248_287.1","last_update":"2021-03-12T02:37:37.578248Z"}] 2021-03-12T02:40:25.945 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-12T02:40:25.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:25.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:25.947 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:25.989 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:25.990 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:26.253 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11911835907256974194},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.617831_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.617831Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615516500.963444_203.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:00.963444Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.578248_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.578248Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:40:26.786 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-12T02:40:26.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:26.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:26.788 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:26.828 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:26.829 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:27.028 INFO:teuthology.orchestra.run.smithi114.stdout:[{"marker":"1_1615516457.862054_318.3","last_update":"2021-03-12T02:34:17.862054Z"},{"marker":"1_1615516820.706047_386.1","last_update":"2021-03-12T02:40:20.706047Z"},{"marker":"1_1615516501.868964_175.1","last_update":"2021-03-12T02:35:01.868964Z"},{"marker":"1_1615516657.657755_238.1","last_update":"2021-03-12T02:37:37.657755Z"}] 2021-03-12T02:40:27.029 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-12T02:40:27.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:27.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:27.030 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:27.073 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:27.073 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:27.289 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9936002905845503227},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516457.862054_318.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:17.862054Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.706047_386.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.706047Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615516501.868964_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:01.868964Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.657755_238.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.657755Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:40:27.290 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-12T02:40:27.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:27.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:27.292 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:27.332 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:27.332 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:27.539 INFO:teuthology.orchestra.run.smithi114.stdout:[{"marker":"1_1615516457.862054_318.3","last_update":"2021-03-12T02:34:17.862054Z"},{"marker":"1_1615516820.706047_386.1","last_update":"2021-03-12T02:40:20.706047Z"},{"marker":"1_1615516501.868964_175.1","last_update":"2021-03-12T02:35:01.868964Z"},{"marker":"1_1615516657.657755_238.1","last_update":"2021-03-12T02:37:37.657755Z"}] 2021-03-12T02:40:27.540 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-12T02:40:27.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:27.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:27.541 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:27.583 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:27.583 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:27.793 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3923957601198887203},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516457.862054_318.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:17.862054Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.706047_386.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.706047Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615516501.868964_175.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:01.868964Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.657755_238.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.657755Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:40:27.793 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-12T02:40:27.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:27.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:27.795 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:27.873 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:27.874 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:28.074 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516456.985100_243.3","last_update":"2021-03-12T02:34:16.985100Z"},{"marker":"1_1615516820.422370_306.1","last_update":"2021-03-12T02:40:20.422370Z"},{"marker":"1_1615516500.991879_145.1","last_update":"2021-03-12T02:35:00.991879Z"},{"marker":"1_1615516657.308107_198.1","last_update":"2021-03-12T02:37:37.308107Z"}] 2021-03-12T02:40:28.075 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-12T02:40:28.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:28.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:28.076 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:28.156 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:28.157 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:28.367 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12616054323329998860},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.985100_243.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.985100Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.422370_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.422370Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615516500.991879_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:00.991879Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.308107_198.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.308107Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:40:28.368 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-12T02:40:28.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:28.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:28.369 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:28.448 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:28.448 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:28.653 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615516456.985100_243.3","last_update":"2021-03-12T02:34:16.985100Z"},{"marker":"1_1615516820.422370_306.1","last_update":"2021-03-12T02:40:20.422370Z"},{"marker":"1_1615516500.991879_145.1","last_update":"2021-03-12T02:35:00.991879Z"},{"marker":"1_1615516657.308107_198.1","last_update":"2021-03-12T02:37:37.308107Z"}] 2021-03-12T02:40:28.654 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-12T02:40:28.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:28.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:28.655 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:28.698 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:28.698 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:28.922 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10754379456686351807},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.985100_243.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.985100Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.422370_306.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.422370Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615516500.991879_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:35:00.991879Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.308107_198.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.308107Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:40:28.923 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-12T02:40:28.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:28.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:28.925 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:40:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:28.961 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:29.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:29.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:29.175 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:29.252 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:29.253 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:29.469 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2021-03-12T02:40:29.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:29.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:29.471 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:40:29.508 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:29.509 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:29.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:29.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:29.709 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:29.787 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:29.787 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:29.998 INFO:teuthology.orchestra.run.smithi114.stdout:[] 2021-03-12T02:40:29.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:29.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:30.000 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:40:30.036 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:30.037 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:30.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:30.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:40:30.264 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:40:30.340 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:30.341 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:30.554 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2021-03-12T02:40:30.555 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2021-03-12T02:40:30.556 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:30.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:30.557 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:40:30.596 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:30.597 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:30.623 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:40:30.620+0000 7ff2ed180080 0 failed reading obj info from .rgw.root:zone_info.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e: (2) No such file or directory 2021-03-12T02:40:30.624 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:40:30.620+0000 7ff2ed180080 0 WARNING: could not read zone params for zone id=6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e name=test-zone1 2021-03-12T02:40:30.624 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:40:30.624+0000 7ff2ed180080 0 failed reading obj info from .rgw.root:zone_info.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107: (2) No such file or directory 2021-03-12T02:40:30.624 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:40:30.624+0000 7ff2ed180080 0 WARNING: could not read zone params for zone id=e98c2a8b-a28f-47cd-a0ca-2777b6b4f107 name=test-zone3 2021-03-12T02:40:30.635 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"IMFN8A2KI1WM97OP","secret_key":"SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"0428d020-efe8-4edc-a347-2def7c927539"} 2021-03-12T02:40:30.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:30.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:40:30.637 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:40:30.718 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:40:30.718 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:40:30.919 INFO:teuthology.orchestra.run.smithi114.stderr:Sending period to new master zone 6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e 2021-03-12T02:40:32.008 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","epoch":2,"predecessor_uuid":"76eba794-fdc5-4e5e-9ce6-5f5a18a97795","sync_status":["","","1_1615516729.491953_32.1",""],"period_map":{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","zones":[{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","endpoints":["http://smithi114.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":""},{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","name":"test-zone4","endpoints":["http://smithi114.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","name":"test-zone3","endpoints":["http://smithi077.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"374a14e6-b245-4495-acdf-8a1b3996ba1d","val":3185083755},{"key":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","val":2537029094},{"key":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","val":3794530578},{"key":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","val":3019679815}]},"master_zonegroup":"test-zonegroup","master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"0428d020-efe8-4edc-a347-2def7c927539","realm_name":"test-realm","realm_epoch":4} 2021-03-12T02:40:32.009 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2021-03-12T02:41:32.013 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-49 2021-03-12T02:41:32.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:41:32.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:41:32.113 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:41:32.153 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:41:32.153 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:41:32.336 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:41:32.367 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516892.057265_686.1","last_update":"2021-03-12T02:41:32.057265Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:41:32.368 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:41:32.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:41:32.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:41:32.369 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:41:32.410 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:41:32.411 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:41:32.596 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516818.948465_625.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:18.948465Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:41:32.596 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615516892.057265_686.1 target=1_1615516818.948465_625.1 2021-03-12T02:41:37.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:41:37.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:41:37.599 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:41:37.641 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:41:37.641 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:41:37.843 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516818.948465_625.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:18.948465Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:41:37.843 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615516892.057265_686.1 target=1_1615516818.948465_625.1 2021-03-12T02:41:42.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:41:42.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:41:42.846 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:41:42.887 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:41:42.888 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:41:43.092 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516818.948465_625.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:18.948465Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:41:43.092 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615516892.057265_686.1 target=1_1615516818.948465_625.1 2021-03-12T02:41:48.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:41:48.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:41:48.095 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:41:48.138 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:41:48.138 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:41:48.339 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516818.948465_625.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:18.948465Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:41:48.339 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615516892.057265_686.1 target=1_1615516818.948465_625.1 2021-03-12T02:41:53.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:41:53.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:41:53.342 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:41:53.381 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:41:53.382 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:41:53.582 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516818.948465_625.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:18.948465Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:41:53.583 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615516892.057265_686.1 target=1_1615516818.948465_625.1 2021-03-12T02:41:58.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:41:58.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:41:58.586 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:41:58.627 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:41:58.627 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:41:58.827 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516818.948465_625.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:18.948465Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:41:58.828 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615516892.057265_686.1 target=1_1615516818.948465_625.1 2021-03-12T02:42:03.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:03.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:03.830 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:42:03.871 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:42:03.871 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:42:04.080 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516892.057265_686.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:41:32.057265Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:42:04.081 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:42:04.081 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:42:04.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:04.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:04.083 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:42:04.123 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:42:04.124 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:42:04.331 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516892.057265_686.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:41:32.057265Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:42:04.332 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:42:04.332 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:42:04.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:04.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:04.334 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:42:04.374 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:42:04.374 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:42:04.638 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516818.948465_625.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:18.948465Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:42:04.639 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1615516892.057265_686.1 target=1_1615516818.948465_625.1 2021-03-12T02:42:09.640 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:09.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:09.641 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:42:09.680 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:42:09.681 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:42:09.947 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516818.948465_625.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:18.948465Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:42:09.948 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1615516892.057265_686.1 target=1_1615516818.948465_625.1 2021-03-12T02:42:14.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:14.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:14.951 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:42:14.994 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:42:14.995 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:42:15.256 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516818.948465_625.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:18.948465Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:42:15.258 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1615516892.057265_686.1 target=1_1615516818.948465_625.1 2021-03-12T02:42:20.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:20.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:20.262 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:42:20.305 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:42:20.306 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:42:20.574 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516818.948465_625.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:18.948465Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:42:20.575 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1615516892.057265_686.1 target=1_1615516818.948465_625.1 2021-03-12T02:42:25.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:25.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:25.578 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:42:25.618 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:42:25.619 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:42:25.879 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516818.948465_625.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:18.948465Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:42:25.880 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1615516892.057265_686.1 target=1_1615516818.948465_625.1 2021-03-12T02:42:30.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:30.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:30.883 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:42:30.924 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:42:30.924 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:42:31.187 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516818.948465_625.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:18.948465Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:42:31.187 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1615516892.057265_686.1 target=1_1615516818.948465_625.1 2021-03-12T02:42:36.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:36.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:42:36.190 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:42:36.231 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:42:36.231 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:42:36.495 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516892.057265_686.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:41:32.057265Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:42:36.496 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:42:36.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:42:36.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:42:36.513 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:42:36.578 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:42:36.578 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:42:36.606 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:42:36.603+0000 7fa352090080 0 failed reading obj info from .rgw.root:zone_info.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e: (2) No such file or directory 2021-03-12T02:42:36.607 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:42:36.603+0000 7fa352090080 0 WARNING: could not read zone params for zone id=6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e name=test-zone1 2021-03-12T02:42:36.607 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:42:36.607+0000 7fa352090080 0 failed reading obj info from .rgw.root:zone_info.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107: (2) No such file or directory 2021-03-12T02:42:36.607 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:42:36.607+0000 7fa352090080 0 WARNING: could not read zone params for zone id=e98c2a8b-a28f-47cd-a0ca-2777b6b4f107 name=test-zone3 2021-03-12T02:42:36.619 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"IMFN8A2KI1WM97OP","secret_key":"SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"0428d020-efe8-4edc-a347-2def7c927539"} 2021-03-12T02:42:36.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:42:36.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:42:36.621 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:42:36.701 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:42:36.702 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:42:36.928 INFO:teuthology.orchestra.run.smithi114.stderr:Sending period to new master zone 6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e 2021-03-12T02:42:38.013 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","epoch":3,"predecessor_uuid":"76eba794-fdc5-4e5e-9ce6-5f5a18a97795","sync_status":["","","1_1615516729.491953_32.1",""],"period_map":{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","zones":[{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","endpoints":["http://smithi114.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e"},{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","name":"test-zone4","endpoints":["http://smithi114.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","name":"test-zone3","endpoints":["http://smithi077.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"374a14e6-b245-4495-acdf-8a1b3996ba1d","val":3185083755},{"key":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","val":2537029094},{"key":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","val":3794530578},{"key":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","val":3019679815}]},"master_zonegroup":"test-zonegroup","master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"0428d020-efe8-4edc-a347-2def7c927539","realm_name":"test-realm","realm_epoch":4} 2021-03-12T02:42:38.268 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e" 2021-03-12T02:43:38.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:43:38.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:43:38.604 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:43:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:43:38.647 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:43:38.673 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:43:38.669+0000 7f81d5927080 0 failed reading obj info from .rgw.root:zone_info.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e: (2) No such file or directory 2021-03-12T02:43:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:43:38.669+0000 7f81d5927080 0 WARNING: could not read zone params for zone id=6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e name=test-zone1 2021-03-12T02:43:38.674 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:43:38.673+0000 7f81d5927080 0 failed reading obj info from .rgw.root:zone_info.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107: (2) No such file or directory 2021-03-12T02:43:38.675 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:43:38.673+0000 7f81d5927080 0 WARNING: could not read zone params for zone id=e98c2a8b-a28f-47cd-a0ca-2777b6b4f107 name=test-zone3 2021-03-12T02:43:38.686 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"IMFN8A2KI1WM97OP","secret_key":"SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"0428d020-efe8-4edc-a347-2def7c927539"} 2021-03-12T02:43:38.687 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:43:38.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:43:38.688 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:43:38.769 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:43:38.770 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:43:38.982 INFO:teuthology.orchestra.run.smithi114.stderr:Sending period to new master zone 6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e 2021-03-12T02:43:40.065 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","epoch":4,"predecessor_uuid":"76eba794-fdc5-4e5e-9ce6-5f5a18a97795","sync_status":["","","1_1615516729.491953_32.1",""],"period_map":{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","zones":[{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","endpoints":["http://smithi114.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e"},{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","name":"test-zone4","endpoints":["http://smithi114.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","name":"test-zone3","endpoints":["http://smithi077.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"374a14e6-b245-4495-acdf-8a1b3996ba1d","val":3185083755},{"key":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","val":2537029094},{"key":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","val":3794530578},{"key":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","val":3019679815}]},"master_zonegroup":"test-zonegroup","master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"0428d020-efe8-4edc-a347-2def7c927539","realm_name":"test-realm","realm_epoch":4} 2021-03-12T02:43:40.066 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2021-03-12T02:44:40.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:44:40.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:44:40.069 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:44:40.110 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:44:40.111 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:44:40.140 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:44:40.140+0000 7f9f9dca6080 0 failed reading obj info from .rgw.root:zone_info.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e: (2) No such file or directory 2021-03-12T02:44:40.140 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:44:40.140+0000 7f9f9dca6080 0 WARNING: could not read zone params for zone id=6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e name=test-zone1 2021-03-12T02:44:40.141 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:44:40.140+0000 7f9f9dca6080 0 failed reading obj info from .rgw.root:zone_info.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107: (2) No such file or directory 2021-03-12T02:44:40.141 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:44:40.140+0000 7f9f9dca6080 0 WARNING: could not read zone params for zone id=e98c2a8b-a28f-47cd-a0ca-2777b6b4f107 name=test-zone3 2021-03-12T02:44:40.153 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"IMFN8A2KI1WM97OP","secret_key":"SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"0428d020-efe8-4edc-a347-2def7c927539"} 2021-03-12T02:44:40.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:44:40.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:44:40.154 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:44:40.235 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:44:40.236 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:44:40.447 INFO:teuthology.orchestra.run.smithi114.stderr:Sending period to new master zone 6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e 2021-03-12T02:44:41.519 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","epoch":5,"predecessor_uuid":"76eba794-fdc5-4e5e-9ce6-5f5a18a97795","sync_status":["","","1_1615516729.491953_32.1",""],"period_map":{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","zones":[{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","endpoints":["http://smithi114.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","name":"test-zone4","endpoints":["http://smithi114.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","name":"test-zone3","endpoints":["http://smithi077.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"374a14e6-b245-4495-acdf-8a1b3996ba1d","val":3185083755},{"key":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","val":2537029094},{"key":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","val":3794530578},{"key":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","val":3019679815}]},"master_zonegroup":"test-zonegroup","master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"0428d020-efe8-4edc-a347-2def7c927539","realm_name":"test-realm","realm_epoch":4} 2021-03-12T02:44:41.520 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2021-03-12T02:45:41.522 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2021-03-12T02:45:41.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:45:41.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:45:41.525 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:45:41.564 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:45:41.564 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:45:41.749 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:45:41.781 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615516892.057265_686.1","last_update":"2021-03-12T02:41:32.057265Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:45:41.782 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:45:41.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:45:41.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:45:41.784 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:45:41.826 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:45:41.827 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:45:42.035 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516892.057265_686.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:41:32.057265Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:45:42.035 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:45:42.036 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:45:42.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:45:42.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:45:42.037 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:45:42.077 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:45:42.078 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:45:42.285 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516892.057265_686.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:41:32.057265Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:45:42.286 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:45:42.287 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:45:42.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:45:42.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:45:42.288 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:45:42.330 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:45:42.330 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:45:42.594 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615516892.057265_686.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:41:32.057265Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:45:42.595 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:45:42.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:42.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'IMFN8A2KI1WM97OP', '--secret', 'SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:42.597 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key IMFN8A2KI1WM97OP --secret SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:45:42.675 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:45:42.676 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:45:42.736 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:45:42.734+0000 7f4dce524080 0 failed reading obj info from .rgw.root:zone_info.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e: (2) No such file or directory 2021-03-12T02:45:42.736 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:45:42.734+0000 7f4dce524080 0 WARNING: could not read zone params for zone id=6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e name=test-zone1 2021-03-12T02:45:42.737 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:45:42.734+0000 7f4dce524080 0 failed reading obj info from .rgw.root:zone_info.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107: (2) No such file or directory 2021-03-12T02:45:42.737 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:45:42.734+0000 7f4dce524080 0 WARNING: could not read zone params for zone id=e98c2a8b-a28f-47cd-a0ca-2777b6b4f107 name=test-zone3 2021-03-12T02:45:42.748 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"01cbe2e1-954a-45dd-8bf4-b8cfbcf4c61e","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"IMFN8A2KI1WM97OP","secret_key":"SKHHGUZ8SdIcaru7oU4ZbCF9ROHYXPLE"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"0428d020-efe8-4edc-a347-2def7c927539"} 2021-03-12T02:45:42.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:42.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:42.749 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:45:42.830 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:45:42.830 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:45:50.925 INFO:teuthology.orchestra.run.smithi114.stderr:Sending period to new master zone 6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e 2021-03-12T02:45:52.009 INFO:teuthology.orchestra.run.smithi114.stdout:{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","epoch":6,"predecessor_uuid":"76eba794-fdc5-4e5e-9ce6-5f5a18a97795","sync_status":["","","1_1615516729.491953_32.1",""],"period_map":{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","zones":[{"id":"01cbe2e1-954a-45dd-8bf4-b8cfbcf4c61e","name":"remove","endpoints":[],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","endpoints":["http://smithi114.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","name":"test-zone4","endpoints":["http://smithi114.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","name":"test-zone3","endpoints":["http://smithi077.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"01cbe2e1-954a-45dd-8bf4-b8cfbcf4c61e","val":180093089},{"key":"374a14e6-b245-4495-acdf-8a1b3996ba1d","val":3185083755},{"key":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","val":2537029094},{"key":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","val":3794530578},{"key":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","val":3019679815}]},"master_zonegroup":"test-zonegroup","master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"0428d020-efe8-4edc-a347-2def7c927539","realm_name":"test-realm","realm_epoch":4} 2021-03-12T02:45:52.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:52.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:52.011 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:45:52.053 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:45:52.053 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:45:52.072 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:45:52.065+0000 7ff9ef9fb080 0 failed reading obj info from .rgw.root:zone_info.374a14e6-b245-4495-acdf-8a1b3996ba1d: (2) No such file or directory 2021-03-12T02:45:52.073 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:45:52.065+0000 7ff9ef9fb080 0 WARNING: could not read zone params for zone id=374a14e6-b245-4495-acdf-8a1b3996ba1d name=test-zone2 2021-03-12T02:45:52.074 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:45:52.069+0000 7ff9ef9fb080 0 failed reading obj info from .rgw.root:zone_info.ad4e1236-7627-4342-9dd1-46e3c42d2e72: (2) No such file or directory 2021-03-12T02:45:52.074 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:45:52.069+0000 7ff9ef9fb080 0 WARNING: could not read zone params for zone id=ad4e1236-7627-4342-9dd1-46e3c42d2e72 name=test-zone4 2021-03-12T02:45:52.082 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","zones":[{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","endpoints":["http://smithi114.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","name":"test-zone4","endpoints":["http://smithi114.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","name":"test-zone3","endpoints":["http://smithi077.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}} 2021-03-12T02:45:52.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:52.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:52.084 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:45:52.164 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:45:52.164 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:45:52.182 INFO:teuthology.orchestra.run.smithi077.stderr:zone name remove not found in zonegroup test-zonegroup 2021-03-12T02:45:52.186 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-03-12T02:45:52.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:52.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:52.189 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-12T02:45:52.225 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:45:52.226 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:45:52.272 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:45:52.270+0000 7fa2d2f6e080 0 failed reading obj info from .rgw.root:zone_info.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e: (2) No such file or directory 2021-03-12T02:45:52.272 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:45:52.270+0000 7fa2d2f6e080 0 WARNING: could not read zone params for zone id=6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e name=test-zone1 2021-03-12T02:45:52.274 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:45:52.270+0000 7fa2d2f6e080 0 failed reading obj info from .rgw.root:zone_info.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107: (2) No such file or directory 2021-03-12T02:45:52.274 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:45:52.270+0000 7fa2d2f6e080 0 WARNING: could not read zone params for zone id=e98c2a8b-a28f-47cd-a0ca-2777b6b4f107 name=test-zone3 2021-03-12T02:45:52.277 INFO:teuthology.orchestra.run.smithi114.stderr:2021-03-12T02:45:52.274+0000 7fa2d2f6e080 0 zone id 01cbe2e1-954a-45dd-8bf4-b8cfbcf4c61e is not a part of zonegroup default 2021-03-12T02:45:52.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:52.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:52.291 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:45:52.332 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:45:52.332 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:45:52.393 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:45:52.389+0000 7f407b82c080 0 WARNING: can't generate connection for zone 01cbe2e1-954a-45dd-8bf4-b8cfbcf4c61e id remove: no endpoints defined 2021-03-12T02:45:52.628 INFO:tasks.rgw.c2.client.1.smithi114.stdout:2021-03-12T02:45:52.626+0000 7fe1a8ff9700 -1 failed to decode period 2021-03-12T02:45:52.628 INFO:tasks.rgw.c2.client.0.smithi114.stdout:2021-03-12T02:45:52.626+0000 7f35a27fc700 -1 failed to decode period 2021-03-12T02:45:52.629 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:45:52.621+0000 7efe20ff9700 -1 failed to decode period 2021-03-12T02:45:52.662 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","epoch":7,"predecessor_uuid":"76eba794-fdc5-4e5e-9ce6-5f5a18a97795","sync_status":["","","1_1615516729.491953_32.1",""],"period_map":{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","zones":[{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","endpoints":["http://smithi114.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","name":"test-zone4","endpoints":["http://smithi114.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","name":"test-zone3","endpoints":["http://smithi077.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"374a14e6-b245-4495-acdf-8a1b3996ba1d","val":3185083755},{"key":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","val":2537029094},{"key":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","val":3794530578},{"key":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","val":3019679815}]},"master_zonegroup":"test-zonegroup","master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"0428d020-efe8-4edc-a347-2def7c927539","realm_name":"test-realm","realm_epoch":4} 2021-03-12T02:45:52.663 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2021-03-12T02:45:52.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:52.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:52.665 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:45:52.739 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:45:52.740 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:45:52.769 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"f3b487f9-222b-4137-9aef-8c1dcab26cbd","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}} 2021-03-12T02:45:52.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:52.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:52.770 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:45:52.850 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:45:52.850 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:45:52.924 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"7704b982-19ea-410f-b1e1-7a7858ba1c7b","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"0428d020-efe8-4edc-a347-2def7c927539"} 2021-03-12T02:45:52.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:52.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:52.926 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:45:53.006 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:45:53.006 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:45:53.273 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","epoch":8,"predecessor_uuid":"76eba794-fdc5-4e5e-9ce6-5f5a18a97795","sync_status":["","","1_1615516729.491953_32.1",""],"period_map":{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","zonegroups":[{"id":"f3b487f9-222b-4137-9aef-8c1dcab26cbd","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"7704b982-19ea-410f-b1e1-7a7858ba1c7b","zones":[{"id":"7704b982-19ea-410f-b1e1-7a7858ba1c7b","name":"remove","endpoints":[],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","zones":[{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","endpoints":["http://smithi114.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","name":"test-zone4","endpoints":["http://smithi114.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","name":"test-zone3","endpoints":["http://smithi077.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"374a14e6-b245-4495-acdf-8a1b3996ba1d","val":3185083755},{"key":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","val":2537029094},{"key":"7704b982-19ea-410f-b1e1-7a7858ba1c7b","val":247803542},{"key":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","val":3794530578},{"key":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","val":3019679815}]},"master_zonegroup":"test-zonegroup","master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"0428d020-efe8-4edc-a347-2def7c927539","realm_name":"test-realm","realm_epoch":4} 2021-03-12T02:45:53.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:53.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:53.276 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:45:53.353 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:45:53.354 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:45:53.411 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:45:53.405+0000 7fb3fed11080 0 zone id 7704b982-19ea-410f-b1e1-7a7858ba1c7b is not a part of zonegroup test-zonegroup 2021-03-12T02:45:53.412 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:45:53.405+0000 7fb3fed11080 0 zone id 7704b982-19ea-410f-b1e1-7a7858ba1c7b is not a part of zonegroup default 2021-03-12T02:45:53.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:53.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:53.426 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:45:53.505 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:45:53.506 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:45:53.560 INFO:teuthology.orchestra.run.smithi077.stderr:2021-03-12T02:45:53.553+0000 7f51ca9d4080 0 ERROR: zonegroup remove_zg has a non existent master zone 2021-03-12T02:45:53.570 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-03-12T02:45:53.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:53.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:53.572 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:45:53.649 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:45:53.650 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:45:53.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:53.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:45:53.685 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:45:53.764 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:45:53.764 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:45:54.066 INFO:teuthology.orchestra.run.smithi077.stdout:{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","epoch":9,"predecessor_uuid":"76eba794-fdc5-4e5e-9ce6-5f5a18a97795","sync_status":["","","1_1615516729.491953_32.1",""],"period_map":{"id":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","zones":[{"id":"374a14e6-b245-4495-acdf-8a1b3996ba1d","name":"test-zone2","endpoints":["http://smithi114.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","name":"test-zone1","endpoints":["http://smithi077.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","name":"test-zone4","endpoints":["http://smithi114.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","name":"test-zone3","endpoints":["http://smithi077.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"0428d020-efe8-4edc-a347-2def7c927539","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"374a14e6-b245-4495-acdf-8a1b3996ba1d","val":3185083755},{"key":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","val":2537029094},{"key":"ad4e1236-7627-4342-9dd1-46e3c42d2e72","val":3794530578},{"key":"e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","val":3019679815}]},"master_zonegroup":"test-zonegroup","master_zone":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"0428d020-efe8-4edc-a347-2def7c927539","realm_name":"test-realm","realm_epoch":4} 2021-03-12T02:45:54.066 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2021-03-12T02:45:54.067 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-50 2021-03-12T02:45:57.798 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-51 2021-03-12T02:46:03.415 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-52 2021-03-12T02:46:04.030 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-53 2021-03-12T02:46:04.068 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-54 2021-03-12T02:46:04.126 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-55 2021-03-12T02:46:06.486 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2021-03-12T02:46:06.488 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-56 2021-03-12T02:46:07.450 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-57 2021-03-12T02:46:07.657 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-58 2021-03-12T02:46:07.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'bbucnh-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:46:07.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'bbucnh-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:46:07.905 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket bbucnh-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:46:07.944 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:07.944 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:08.199 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'bbucnh-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:46:08.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'bbucnh-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:46:08.200 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket bbucnh-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:46:08.279 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:08.280 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:08.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'bbucnh-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:46:08.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'bbucnh-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:46:08.552 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket bbucnh-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:46:08.633 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:08.634 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:08.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:08.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:08.907 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:46:08.985 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:08.985 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:09.234 INFO:teuthology.orchestra.run.smithi077.stdout:["obj_delete_at_hint.0000000078","meta.history","datalog.sync-status.shard.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.2","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:8","obj_delete_at_hint.0000000070","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:8","obj_delete_at_hint.0000000104","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.6-bbucnh-6","obj_delete_at_hint.0000000026","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31-bbucnh-22","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:1","obj_delete_at_hint.0000000028","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:1","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-14:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23:1","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-37:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46:10","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.5-bbucnh-5","datalog.sync-status.shard.374a14e6-b245-4495-acdf-8a1b3996ba1d.3","data.full-sync.index.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.1","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:3","data.full-sync.index.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.0","obj_delete_at_hint.0000000040","meta.full-sync.index.0","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:3","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38-bbucnh-29","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-48:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.5:8","data.full-sync.index.374a14e6-b245-4495-acdf-8a1b3996ba1d.1","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-49:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6042.1:5","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","datalog.sync-status.shard.374a14e6-b245-4495-acdf-8a1b3996ba1d.3.retry","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.48-bbucnh-39","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35:8","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.1-bbucnh-1","datalog.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22-bbucnh-13","data_log.3","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:3","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-13:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22:1","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-47:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.4:8","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32-bbucnh-23","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34:8","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46-bbucnh-37","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:1","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","data.full-sync.index.374a14e6-b245-4495-acdf-8a1b3996ba1d.3","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.4-bbucnh-4","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:5","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.50-bbucnh-41","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29:8","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35:8","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28:8","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:8","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:5","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:8","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34:8","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41-bbucnh-32","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:7","obj_delete_at_hint.0000000014","datalog.sync-status.shard.374a14e6-b245-4495-acdf-8a1b3996ba1d.0.retry","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-13:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22:1","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32:8","obj_delete_at_hint.0000000112","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:1","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","datalog.sync-status.shard.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.3","obj_delete_at_hint.0000000071","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-38:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47:10","obj_delete_at_hint.0000000074","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.3-bbucnh-52","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.1-bbucnh-44","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24-bbucnh-15","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.42-bbucnh-33","data_log.1","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:8","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28:8","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-15:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24:1","obj_delete_at_hint.0000000125","2021-03-12-02-374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.1-bbucnh-42","obj_delete_at_hint.0000000082","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.49-bbucnh-40","meta.full-sync.index.1","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45-bbucnh-36","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:5","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31:8","obj_delete_at_hint.0000000077","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.16-bbucnh-7","obj_delete_at_hint.0000000032","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19-bbucnh-10","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20-bbucnh-11","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:3","data.full-sync.index.374a14e6-b245-4495-acdf-8a1b3996ba1d.2","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:1","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.39-bbucnh-30","obj_delete_at_hint.0000000053","data.full-sync.index.374a14e6-b245-4495-acdf-8a1b3996ba1d.0","obj_delete_at_hint.0000000091","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.2-bbucnh-51","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40-bbucnh-31","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29:8","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:1","mdlog.sync-status.shard.2","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-14:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23:1","datalog.sync-status.shard.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.3.retry","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31:8","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","obj_delete_at_hint.0000000116","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.2-bbucnh-45","mdlog.sync-status.shard.1","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.6-bbucnh-55","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.3-bbucnh-3","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:1","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:7","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.8-bbucnh-57","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.4-bbucnh-47","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-38:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47:10","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","meta.full-sync.index.3","obj_delete_at_hint.0000000118","datalog.sync-status.shard.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.1.retry","obj_delete_at_hint.0000000036","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-37:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46:10","obj_delete_at_hint.0000000103","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:1","mdlog.sync-status.shard.3","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23-bbucnh-14","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-49:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6042.1:5","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28:5","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:7","obj_delete_at_hint.0000000039","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:1","datalog.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29-bbucnh-20","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:7","obj_delete_at_hint.0000000100","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:8","data.full-sync.index.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.2","obj_delete_at_hint.0000000093","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37-bbucnh-28","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","datalog.sync-status.shard.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.2.retry","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:3","obj_delete_at_hint.0000000057","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25-bbucnh-16","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26-bbucnh-17","data_log.0","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.5-bbucnh-48","data.full-sync.index.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.3","obj_delete_at_hint.0000000012","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:5","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.4-bbucnh-53","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:1","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","2021-03-12-02-374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.2-bbucnh-43","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43-bbucnh-34","obj_delete_at_hint.0000000020","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:3","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:3","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:3","datalog.sync-status.shard.374a14e6-b245-4495-acdf-8a1b3996ba1d.1","datalog.sync-status.shard.374a14e6-b245-4495-acdf-8a1b3996ba1d.1.retry","obj_delete_at_hint.0000000072","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21-bbucnh-12","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","sync.error-log.1","obj_delete_at_hint.0000000033","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.18-bbucnh-9","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34-bbucnh-25","obj_delete_at_hint.0000000025","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:7","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.5-bbucnh-54","obj_delete_at_hint.0000000123","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:7","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:3","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","meta.log.8d2c0f1d-7035-4313-b22b-345f1f4f81db.2","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-47:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.4:8","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6042.1-bbucnh-49","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.36-bbucnh-27","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30-bbucnh-21","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33-bbucnh-24","obj_delete_at_hint.0000000051","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47-bbucnh-38","obj_delete_at_hint.0000000106","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.7-bbucnh-56","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:1","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:1","obj_delete_at_hint.0000000017","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:3","obj_delete_at_hint.0000000023","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-46:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.3:8","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:5","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28-bbucnh-19","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.2-bbucnh-2","meta.full-sync.index.2","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30:8","datalog.sync-status.shard.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.0","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-15:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24:1","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.9-bbucnh-58","data_log.2","obj_delete_at_hint.0000000067","datalog.sync-status.shard.374a14e6-b245-4495-acdf-8a1b3996ba1d.0","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33:8","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-46:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.3:8","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28:5","datalog.sync-status.shard.374a14e6-b245-4495-acdf-8a1b3996ba1d.2.retry","obj_delete_at_hint.0000000086","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27-bbucnh-18","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:5","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35-bbucnh-26","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30:8","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:3","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","datalog.sync-status.shard.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.0.retry","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.17-bbucnh-8","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.1-bbucnh-50","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33:8","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:3","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32:8","obj_delete_at_hint.0000000126","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-36:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45:10","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","datalog.sync-status.shard.374a14e6-b245-4495-acdf-8a1b3996ba1d.2","obj_delete_at_hint.0000000097","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44-bbucnh-35","obj_delete_at_hint.0000000042","datalog.sync-status.shard.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.1","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.3-bbucnh-46","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-48:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.5:8","obj_delete_at_hint.0000000075","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-36:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45:10","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030"] 2021-03-12T02:46:09.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:09.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:09.254 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:46:09.295 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:09.295 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:09.479 INFO:teuthology.orchestra.run.smithi114.stdout:["obj_delete_at_hint.0000000078","meta.history","datalog.sync-status.shard.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.2","obj_delete_at_hint.0000000070","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:8","obj_delete_at_hint.0000000104","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.6-bbucnh-6","obj_delete_at_hint.0000000026","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-14:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23:1","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31-bbucnh-22","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:1","obj_delete_at_hint.0000000028","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-14:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23:1","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.5-bbucnh-5","data.full-sync.index.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.1","data.full-sync.index.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.0","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-49:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6042.1:5","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.2","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","obj_delete_at_hint.0000000024","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:7","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.48-bbucnh-39","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35:8","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.1-bbucnh-1","datalog.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22-bbucnh-13","data_log.3","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-13:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22:1","data.full-sync.index.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.1","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-37:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46:10","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32-bbucnh-23","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34:8","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46-bbucnh-37","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:1","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:8","obj_delete_at_hint.0000000090","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35:8","obj_delete_at_hint.0000000110","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.2.retry","obj_delete_at_hint.0000000008","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.4-bbucnh-4","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:5","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.50-bbucnh-41","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29:8","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:1","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:8","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:5","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:3","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:8","obj_delete_at_hint.0000000014","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:5","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32:8","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","datalog.sync-status.shard.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.3","obj_delete_at_hint.0000000071","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-38:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47:10","obj_delete_at_hint.0000000074","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.1-bbucnh-44","data.full-sync.index.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.3","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24-bbucnh-15","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28:8","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:7","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-15:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24:1","obj_delete_at_hint.0000000125","2021-03-12-02-374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.1-bbucnh-42","obj_delete_at_hint.0000000082","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.49-bbucnh-40","meta.full-sync.index.1","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45-bbucnh-36","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29:8","obj_delete_at_hint.0000000077","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.16-bbucnh-7","obj_delete_at_hint.0000000032","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19-bbucnh-10","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20-bbucnh-11","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:3","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32:8","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-13:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22:1","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.2-bbucnh-51","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.40-bbucnh-31","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.3.retry","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:1","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:7","mdlog.sync-status.shard.2","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.3","datalog.sync-status.shard.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.3.retry","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31:8","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-47:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.4:8","obj_delete_at_hint.0000000116","obj_delete_at_hint.0000000034","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28:5","obj_delete_at_hint.0000000101","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.2-bbucnh-45","mdlog.sync-status.shard.1","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:1","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:3","data.full-sync.index.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.0","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.3-bbucnh-3","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:1","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:7","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:8","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.8-bbucnh-57","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.4-bbucnh-47","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","meta.full-sync.index.3","obj_delete_at_hint.0000000118","datalog.sync-status.shard.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.1.retry","obj_delete_at_hint.0000000036","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-37:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46:10","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23-bbucnh-14","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-36:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45:10","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:7","obj_delete_at_hint.0000000039","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:1","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-49:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6042.1:5","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29-bbucnh-20","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:7","obj_delete_at_hint.0000000100","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:8","data.full-sync.index.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.2","data.full-sync.index.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.2","obj_delete_at_hint.0000000093","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.0.retry","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.37-bbucnh-28","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","datalog.sync-status.shard.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.2.retry","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:3","obj_delete_at_hint.0000000057","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25-bbucnh-16","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26-bbucnh-17","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31:8","data_log.0","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.5-bbucnh-48","data.full-sync.index.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.3","obj_delete_at_hint.0000000012","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:5","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28:8","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.1.retry","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:3","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:1","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","2021-03-12-02-374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.2-bbucnh-43","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.43-bbucnh-34","obj_delete_at_hint.0000000020","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:3","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:3","obj_delete_at_hint.0000000072","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21-bbucnh-12","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.18-bbucnh-9","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:1","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-46:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.3:8","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34-bbucnh-25","obj_delete_at_hint.0000000025","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.5-bbucnh-54","obj_delete_at_hint.0000000123","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:3","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","meta.log.8d2c0f1d-7035-4313-b22b-345f1f4f81db.2","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-38:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47:10","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-47:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.4:8","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6042.1-bbucnh-49","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30-bbucnh-21","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34:8","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33-bbucnh-24","obj_delete_at_hint.0000000051","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47-bbucnh-38","obj_delete_at_hint.0000000106","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:1","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:1","obj_delete_at_hint.0000000017","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-15:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24:1","obj_delete_at_hint.0000000023","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:5","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28-bbucnh-19","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.2-bbucnh-2","meta.full-sync.index.2","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30:8","datalog.sync-status.shard.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.0","data_log.2","obj_delete_at_hint.0000000067","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-46:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.3:8","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28:5","obj_delete_at_hint.0000000086","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27-bbucnh-18","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35-bbucnh-26","datalog.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:3","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33:8","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","datalog.sync-status.shard.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.0.retry","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.17-bbucnh-8","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:3","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33:8","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:3","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.0","obj_delete_at_hint.0000000124","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30:8","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-36:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45:10","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:1","obj_delete_at_hint.0000000097","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:3","obj_delete_at_hint.0000000042","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-48:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.5:8","datalog.sync-status.shard.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107.1","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.3-bbucnh-46","bucket.sync-status.e98c2a8b-a28f-47cd-a0ca-2777b6b4f107:bbucnh-48:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.5:8","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.1","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:5"] 2021-03-12T02:46:09.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:09.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:09.506 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:46:09.543 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:09.544 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:09.782 INFO:teuthology.orchestra.run.smithi077.stdout:["obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:8","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.6-bbucnh-6","obj_delete_at_hint.0000000026","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-14:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23:1","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31-bbucnh-22","obj_delete_at_hint.0000000028","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:1","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-37:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46:10","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.5-bbucnh-5","datalog.sync-status.shard.374a14e6-b245-4495-acdf-8a1b3996ba1d.3","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:3","obj_delete_at_hint.0000000040","meta.full-sync.index.0","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:3","meta.log.76eba794-fdc5-4e5e-9ce6-5f5a18a97795.2","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.38-bbucnh-29","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-48:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.5:8","data.full-sync.index.374a14e6-b245-4495-acdf-8a1b3996ba1d.1","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.2","datalog.sync-status.shard.374a14e6-b245-4495-acdf-8a1b3996ba1d.3.retry","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","obj_delete_at_hint.0000000024","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:7","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.48-bbucnh-39","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.1-bbucnh-1","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22-bbucnh-13","data_log.3","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:3","data.full-sync.index.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.1","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-47:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.4:8","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-37:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46:10","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32-bbucnh-23","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46-bbucnh-37","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:8","obj_delete_at_hint.0000000090","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35:8","obj_delete_at_hint.0000000110","data.full-sync.index.374a14e6-b245-4495-acdf-8a1b3996ba1d.3","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.2.retry","obj_delete_at_hint.0000000008","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.4-bbucnh-4","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.50-bbucnh-41","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35:8","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:1","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:8","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28:8","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:5","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:8","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:3","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:5","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34:8","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.41-bbucnh-32","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:7","meta.log.0cf3ee11-29e6-4776-941b-50bda62adf82.2","obj_delete_at_hint.0000000014","datalog.sync-status.shard.374a14e6-b245-4495-acdf-8a1b3996ba1d.0.retry","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:5","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-13:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22:1","obj_delete_at_hint.0000000112","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:1","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.3-bbucnh-52","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.1-bbucnh-44","data.full-sync.index.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.3","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24-bbucnh-15","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:8","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:7","meta.log.0cf3ee11-29e6-4776-941b-50bda62adf82.0","obj_delete_at_hint.0000000125","2021-03-12-02-374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.1-bbucnh-42","obj_delete_at_hint.0000000082","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.49-bbucnh-40","meta.full-sync.index.1","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45-bbucnh-36","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:5","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31:8","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29:8","meta.log.0cf3ee11-29e6-4776-941b-50bda62adf82.3","obj_delete_at_hint.0000000077","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.16-bbucnh-7","obj_delete_at_hint.0000000032","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19-bbucnh-10","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20-bbucnh-11","data.full-sync.index.374a14e6-b245-4495-acdf-8a1b3996ba1d.2","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32:8","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:1","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-13:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22:1","obj_delete_at_hint.0000000053","data.full-sync.index.374a14e6-b245-4495-acdf-8a1b3996ba1d.0","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29:8","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.3.retry","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:1","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:7","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:1","mdlog.sync-status.shard.2","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-14:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23:1","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.3","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-47:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.4:8","obj_delete_at_hint.0000000116","obj_delete_at_hint.0000000034","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28:5","obj_delete_at_hint.0000000101","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.2-bbucnh-45","mdlog.sync-status.shard.1","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:1","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:3","data.full-sync.index.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.0","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.6-bbucnh-55","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.3-bbucnh-3","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:8","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.4-bbucnh-47","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-38:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47:10","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","meta.full-sync.index.3","obj_delete_at_hint.0000000118","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:1","mdlog.sync-status.shard.3","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23-bbucnh-14","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-36:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45:10","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-49:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6042.1:5","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28:5","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","obj_delete_at_hint.0000000039","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-49:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6042.1:5","datalog.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29-bbucnh-20","obj_delete_at_hint.0000000100","data.full-sync.index.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.2","obj_delete_at_hint.0000000093","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.0.retry","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25-bbucnh-16","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26-bbucnh-17","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31:8","data_log.0","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.5-bbucnh-48","obj_delete_at_hint.0000000012","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28:8","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.1.retry","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:3","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:1","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:1","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","2021-03-12-02-374a14e6-b245-4495-acdf-8a1b3996ba1d.5431.2-bbucnh-43","obj_delete_at_hint.0000000020","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:3","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","datalog.sync-status.shard.374a14e6-b245-4495-acdf-8a1b3996ba1d.1","datalog.sync-status.shard.374a14e6-b245-4495-acdf-8a1b3996ba1d.1.retry","obj_delete_at_hint.0000000072","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21-bbucnh-12","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.18-bbucnh-9","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:1","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-46:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.3:8","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34-bbucnh-25","obj_delete_at_hint.0000000025","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:7","obj_delete_at_hint.0000000123","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:7","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","meta.log.8d2c0f1d-7035-4313-b22b-345f1f4f81db.2","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-38:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47:10","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6042.1-bbucnh-49","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30-bbucnh-21","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34:8","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33-bbucnh-24","obj_delete_at_hint.0000000051","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47-bbucnh-38","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:3","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-15:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24:1","obj_delete_at_hint.0000000023","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-46:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.3:8","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28-bbucnh-19","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.2-bbucnh-2","meta.full-sync.index.2","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-15:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24:1","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.9-bbucnh-58","data_log.2","obj_delete_at_hint.0000000067","datalog.sync-status.shard.374a14e6-b245-4495-acdf-8a1b3996ba1d.0","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33:8","datalog.sync-status.shard.374a14e6-b245-4495-acdf-8a1b3996ba1d.2.retry","obj_delete_at_hint.0000000086","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27-bbucnh-18","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:5","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35-bbucnh-26","datalog.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30:8","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:3","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:3","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33:8","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.17-bbucnh-8","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:3","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.0","obj_delete_at_hint.0000000124","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30:8","obj_delete_at_hint.0000000001","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32:8","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","datalog.sync-status.shard.374a14e6-b245-4495-acdf-8a1b3996ba1d.2","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:1","obj_delete_at_hint.0000000097","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:3","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.44-bbucnh-35","obj_delete_at_hint.0000000042","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-48:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.5:8","2021-03-12-02-6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.3-bbucnh-46","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.1","obj_delete_at_hint.0000000075","bucket.sync-status.374a14e6-b245-4495-acdf-8a1b3996ba1d:bbucnh-36:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45:10","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:5"] 2021-03-12T02:46:09.808 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:09.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:09.809 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:46:09.846 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:09.847 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:10.048 INFO:teuthology.orchestra.run.smithi114.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-14:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.23:1","obj_delete_at_hint.0000000028","obj_delete_at_hint.0000000040","meta.full-sync.index.0","meta.log.76eba794-fdc5-4e5e-9ce6-5f5a18a97795.2","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.2","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","obj_delete_at_hint.0000000024","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:7","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","data.full-sync.index.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.1","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-37:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.46:10","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:8","obj_delete_at_hint.0000000090","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-26:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.35:8","obj_delete_at_hint.0000000110","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.2.retry","obj_delete_at_hint.0000000008","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:1","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:8","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:5","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:3","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","meta.log.0cf3ee11-29e6-4776-941b-50bda62adf82.2","obj_delete_at_hint.0000000014","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:5","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","data.full-sync.index.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.3","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:7","meta.log.0cf3ee11-29e6-4776-941b-50bda62adf82.0","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-20:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.29:8","meta.log.0cf3ee11-29e6-4776-941b-50bda62adf82.3","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-23:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.32:8","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-13:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.22:1","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.3.retry","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:1","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:7","mdlog.sync-status.shard.2","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.3","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-47:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.4:8","obj_delete_at_hint.0000000116","obj_delete_at_hint.0000000034","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28:5","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:1","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:3","data.full-sync.index.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.0","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:8","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","meta.full-sync.index.3","obj_delete_at_hint.0000000118","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-36:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.45:10","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","obj_delete_at_hint.0000000039","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-49:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6042.1:5","obj_delete_at_hint.0000000100","data.full-sync.index.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.2","obj_delete_at_hint.0000000093","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.0.retry","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-22:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.31:8","obj_delete_at_hint.0000000012","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-19:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.28:8","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.1.retry","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-12:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.21:3","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:1","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:1","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-46:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.3:8","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","meta.log.8d2c0f1d-7035-4313-b22b-345f1f4f81db.2","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-38:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.47:10","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-25:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.34:8","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-15:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.24:1","obj_delete_at_hint.0000000023","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","meta.full-sync.index.2","obj_delete_at_hint.0000000067","obj_delete_at_hint.0000000086","datalog.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-17:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.26:3","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-24:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.33:8","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-18:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.27:3","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.0","obj_delete_at_hint.0000000124","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-21:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.30:8","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-10:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.19:1","obj_delete_at_hint.0000000097","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-11:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.20:3","obj_delete_at_hint.0000000042","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-48:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.5855.5:8","datalog.sync-status.shard.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.1","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","bucket.sync-status.6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-16:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.4342.25:5"] 2021-03-12T02:46:10.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:10.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:10.115 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:46:10.155 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:10.155 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:10.379 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615517168.873462_517.2","last_update":"2021-03-12T02:46:08.873462Z"},{"marker":"1_1615517168.873462_603.3","last_update":"2021-03-12T02:46:08.873462Z"},{"marker":"1_1615517168.873462_524.3","last_update":"2021-03-12T02:46:08.873462Z"},{"marker":"1_1615517168.873462_472.3","last_update":"2021-03-12T02:46:08.873462Z"}] 2021-03-12T02:46:10.379 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-12T02:46:10.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:10.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:10.381 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:46:10.422 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:10.422 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:10.633 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8376578737807985417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.617831_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.617831Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517020.168675_429.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:43:40.168675Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.578248_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.578248Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:46:10.633 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1615517168.873462_517.2 target=1_1615516456.957047_291.3, shard=1 master=1_1615517168.873462_603.3 target=1_1615516820.617831_435.1, shard=2 master=1_1615517168.873462_524.3 target=1_1615517020.168675_429.1, shard=3 master=1_1615517168.873462_472.3 target=1_1615516657.578248_287.1 2021-03-12T02:46:12.784 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:46:12.776+0000 7eff2e7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:46:12.784 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:46:12.776+0000 7eff237f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:46:12.785 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:46:12.776+0000 7eff247f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:46:12.788 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:46:12.776+0000 7eff22ff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:46:12.789 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:46:12.776+0000 7eff2d7fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:46:12.789 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:46:12.776+0000 7eff25ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:46:12.789 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:46:12.780+0000 7eff1f7ee700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:46:12.790 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:46:12.780+0000 7eff21ff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:46:12.790 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:46:12.780+0000 7eff1ffef700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:46:12.790 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:46:12.780+0000 7eff227f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:46:12.791 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:46:12.780+0000 7eff1dfeb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:46:12.791 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:46:12.780+0000 7eff207f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:46:12.791 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:46:12.780+0000 7eff1efed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:46:12.792 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:46:12.780+0000 7eff467fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:46:12.792 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:46:12.780+0000 7eff45ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:46:12.792 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:46:12.780+0000 7eff2ffff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:46:15.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:15.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:15.636 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:46:15.679 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:15.680 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:15.886 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8376578737807985417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.617831_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.617831Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517020.168675_429.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:43:40.168675Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.578248_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.578248Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:46:16.140 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1615517168.873462_517.2 target=1_1615516456.957047_291.3, shard=1 master=1_1615517168.873462_603.3 target=1_1615516820.617831_435.1, shard=2 master=1_1615517168.873462_524.3 target=1_1615517020.168675_429.1, shard=3 master=1_1615517168.873462_472.3 target=1_1615516657.578248_287.1 2021-03-12T02:46:21.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:21.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:21.143 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:46:21.185 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:21.186 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:21.396 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8376578737807985417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.617831_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.617831Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517020.168675_429.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:43:40.168675Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.578248_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.578248Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:46:21.397 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1615517168.873462_517.2 target=1_1615516456.957047_291.3, shard=1 master=1_1615517168.873462_603.3 target=1_1615516820.617831_435.1, shard=2 master=1_1615517168.873462_524.3 target=1_1615517020.168675_429.1, shard=3 master=1_1615517168.873462_472.3 target=1_1615516657.578248_287.1 2021-03-12T02:46:26.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:26.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:26.400 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:46:26.441 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:26.442 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:26.641 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8376578737807985417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.617831_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.617831Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517020.168675_429.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:43:40.168675Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.578248_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.578248Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:46:26.642 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1615517168.873462_517.2 target=1_1615516456.957047_291.3, shard=1 master=1_1615517168.873462_603.3 target=1_1615516820.617831_435.1, shard=2 master=1_1615517168.873462_524.3 target=1_1615517020.168675_429.1, shard=3 master=1_1615517168.873462_472.3 target=1_1615516657.578248_287.1 2021-03-12T02:46:31.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:31.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:31.645 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:46:31.686 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:31.687 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:31.882 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8376578737807985417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.617831_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.617831Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517020.168675_429.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:43:40.168675Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.578248_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.578248Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:46:31.883 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1615517168.873462_517.2 target=1_1615516456.957047_291.3, shard=1 master=1_1615517168.873462_603.3 target=1_1615516820.617831_435.1, shard=2 master=1_1615517168.873462_524.3 target=1_1615517020.168675_429.1, shard=3 master=1_1615517168.873462_472.3 target=1_1615516657.578248_287.1 2021-03-12T02:46:36.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:36.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:36.886 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:46:36.926 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:36.927 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:37.135 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8376578737807985417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.617831_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.617831Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517020.168675_429.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:43:40.168675Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.578248_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.578248Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:46:37.136 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1615517168.873462_517.2 target=1_1615516456.957047_291.3, shard=1 master=1_1615517168.873462_603.3 target=1_1615516820.617831_435.1, shard=2 master=1_1615517168.873462_524.3 target=1_1615517020.168675_429.1, shard=3 master=1_1615517168.873462_472.3 target=1_1615516657.578248_287.1 2021-03-12T02:46:42.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:42.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:42.138 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:46:42.181 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:42.182 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:42.386 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8376578737807985417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.617831_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.617831Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517020.168675_429.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:43:40.168675Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.578248_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.578248Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:46:42.387 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1615517168.873462_517.2 target=1_1615516456.957047_291.3, shard=1 master=1_1615517168.873462_603.3 target=1_1615516820.617831_435.1, shard=2 master=1_1615517168.873462_524.3 target=1_1615517020.168675_429.1, shard=3 master=1_1615517168.873462_472.3 target=1_1615516657.578248_287.1 2021-03-12T02:46:47.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:47.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:47.390 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:46:47.430 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:47.431 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:47.621 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8376578737807985417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.617831_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.617831Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517020.168675_429.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:43:40.168675Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.578248_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.578248Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:46:47.622 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1615517168.873462_517.2 target=1_1615516456.957047_291.3, shard=1 master=1_1615517168.873462_603.3 target=1_1615516820.617831_435.1, shard=2 master=1_1615517168.873462_524.3 target=1_1615517020.168675_429.1, shard=3 master=1_1615517168.873462_472.3 target=1_1615516657.578248_287.1 2021-03-12T02:46:52.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:52.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:52.624 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:46:52.667 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:52.668 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:52.863 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8376578737807985417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.617831_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.617831Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517020.168675_429.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:43:40.168675Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.578248_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.578248Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:46:52.863 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1615517168.873462_517.2 target=1_1615516456.957047_291.3, shard=1 master=1_1615517168.873462_603.3 target=1_1615516820.617831_435.1, shard=2 master=1_1615517168.873462_524.3 target=1_1615517020.168675_429.1, shard=3 master=1_1615517168.873462_472.3 target=1_1615516657.578248_287.1 2021-03-12T02:46:57.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:57.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:46:57.866 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:46:57.908 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:46:57.909 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:46:58.103 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8376578737807985417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.617831_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.617831Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517020.168675_429.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:43:40.168675Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.578248_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.578248Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:46:58.104 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1615517168.873462_517.2 target=1_1615516456.957047_291.3, shard=1 master=1_1615517168.873462_603.3 target=1_1615516820.617831_435.1, shard=2 master=1_1615517168.873462_524.3 target=1_1615517020.168675_429.1, shard=3 master=1_1615517168.873462_472.3 target=1_1615516657.578248_287.1 2021-03-12T02:47:03.105 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:03.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:03.106 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:03.149 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:03.150 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:03.353 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8376578737807985417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.617831_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.617831Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517020.168675_429.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:43:40.168675Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.578248_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.578248Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:47:03.354 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1615517168.873462_517.2 target=1_1615516456.957047_291.3, shard=1 master=1_1615517168.873462_603.3 target=1_1615516820.617831_435.1, shard=2 master=1_1615517168.873462_524.3 target=1_1615517020.168675_429.1, shard=3 master=1_1615517168.873462_472.3 target=1_1615516657.578248_287.1 2021-03-12T02:47:08.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:08.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:08.357 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:08.399 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:08.399 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:08.595 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8376578737807985417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517168.873462_517.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:08.873462Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517168.873462_603.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:08.873462Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517168.873462_524.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:08.873462Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517168.873462_472.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:08.873462Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:47:08.595 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-12T02:47:08.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:08.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:08.597 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:08.637 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:08.858 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615517168.873462_517.2","last_update":"2021-03-12T02:46:08.873462Z"},{"marker":"1_1615517168.873462_603.3","last_update":"2021-03-12T02:46:08.873462Z"},{"marker":"1_1615517168.873462_524.3","last_update":"2021-03-12T02:46:08.873462Z"},{"marker":"1_1615517168.873462_472.3","last_update":"2021-03-12T02:46:08.873462Z"}] 2021-03-12T02:47:08.859 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-12T02:47:08.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:08.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:08.861 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:08.942 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:08.943 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:09.153 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1279455150295362088},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517168.873462_517.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:08.873462Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517168.873462_603.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:08.873462Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517168.873462_524.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:08.873462Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517168.873462_472.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:08.873462Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:47:09.153 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-12T02:47:09.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:09.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:09.155 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:09.231 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:09.231 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:09.443 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615517168.873462_517.2","last_update":"2021-03-12T02:46:08.873462Z"},{"marker":"1_1615517168.873462_603.3","last_update":"2021-03-12T02:46:08.873462Z"},{"marker":"1_1615517168.873462_524.3","last_update":"2021-03-12T02:46:08.873462Z"},{"marker":"1_1615517168.873462_472.3","last_update":"2021-03-12T02:46:08.873462Z"}] 2021-03-12T02:47:09.444 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-12T02:47:09.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:09.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:09.445 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:09.488 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:09.488 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:09.749 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11911835907256974194},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516456.957047_291.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:16.957047Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.617831_435.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.617831Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517020.168675_429.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:43:40.168675Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.578248_287.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.578248Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:47:09.750 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1615517168.873462_517.2 target=1_1615516456.957047_291.3, shard=1 master=1_1615517168.873462_603.3 target=1_1615516820.617831_435.1, shard=2 master=1_1615517168.873462_524.3 target=1_1615517020.168675_429.1, shard=3 master=1_1615517168.873462_472.3 target=1_1615516657.578248_287.1 2021-03-12T02:47:14.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:14.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:14.753 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:14.797 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:14.797 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:15.068 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11911835907256974194},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517168.873462_517.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:08.873462Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517168.873462_603.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:08.873462Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517168.873462_524.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:08.873462Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517168.873462_472.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:08.873462Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:47:15.068 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-12T02:47:15.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:15.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:15.071 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:15.153 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:15.154 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:15.362 INFO:teuthology.orchestra.run.smithi114.stdout:[{"marker":"1_1615517185.237222_553.2","last_update":"2021-03-12T02:46:25.237222Z"},{"marker":"1_1615517185.237222_550.3","last_update":"2021-03-12T02:46:25.237222Z"},{"marker":"1_1615517185.237222_405.3","last_update":"2021-03-12T02:46:25.237222Z"},{"marker":"1_1615517185.237222_418.3","last_update":"2021-03-12T02:46:25.237222Z"}] 2021-03-12T02:47:15.362 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-12T02:47:15.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:15.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:15.364 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:15.404 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:15.404 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:15.619 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9936002905845503227},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516457.862054_318.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:17.862054Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.706047_386.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.706047Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517043.271340_351.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:44:03.271340Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.657755_238.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.657755Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:47:15.620 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1615517185.237222_553.2 target=1_1615516457.862054_318.3, shard=1 master=1_1615517185.237222_550.3 target=1_1615516820.706047_386.1, shard=2 master=1_1615517185.237222_405.3 target=1_1615517043.271340_351.1, shard=3 master=1_1615517185.237222_418.3 target=1_1615516657.657755_238.1 2021-03-12T02:47:20.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:20.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:20.622 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:20.664 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:20.664 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:20.887 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9936002905845503227},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516457.862054_318.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:17.862054Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.706047_386.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.706047Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517043.271340_351.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:44:03.271340Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.657755_238.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.657755Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:47:20.888 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1615517185.237222_553.2 target=1_1615516457.862054_318.3, shard=1 master=1_1615517185.237222_550.3 target=1_1615516820.706047_386.1, shard=2 master=1_1615517185.237222_405.3 target=1_1615517043.271340_351.1, shard=3 master=1_1615517185.237222_418.3 target=1_1615516657.657755_238.1 2021-03-12T02:47:25.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:25.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:25.891 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:25.932 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:25.933 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:26.153 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9936002905845503227},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516457.862054_318.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:17.862054Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.706047_386.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.706047Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517043.271340_351.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:44:03.271340Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.657755_238.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.657755Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:47:26.153 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1615517185.237222_553.2 target=1_1615516457.862054_318.3, shard=1 master=1_1615517185.237222_550.3 target=1_1615516820.706047_386.1, shard=2 master=1_1615517185.237222_405.3 target=1_1615517043.271340_351.1, shard=3 master=1_1615517185.237222_418.3 target=1_1615516657.657755_238.1 2021-03-12T02:47:31.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:31.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:31.157 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:31.199 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:31.200 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:31.400 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9936002905845503227},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516457.862054_318.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:17.862054Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.706047_386.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.706047Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517043.271340_351.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:44:03.271340Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.657755_238.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.657755Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:47:31.401 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1615517185.237222_553.2 target=1_1615516457.862054_318.3, shard=1 master=1_1615517185.237222_550.3 target=1_1615516820.706047_386.1, shard=2 master=1_1615517185.237222_405.3 target=1_1615517043.271340_351.1, shard=3 master=1_1615517185.237222_418.3 target=1_1615516657.657755_238.1 2021-03-12T02:47:36.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:36.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:36.404 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:36.446 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:36.446 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:36.658 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9936002905845503227},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516457.862054_318.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:17.862054Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.706047_386.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.706047Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517043.271340_351.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:44:03.271340Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.657755_238.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.657755Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:47:36.659 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1615517185.237222_553.2 target=1_1615516457.862054_318.3, shard=1 master=1_1615517185.237222_550.3 target=1_1615516820.706047_386.1, shard=2 master=1_1615517185.237222_405.3 target=1_1615517043.271340_351.1, shard=3 master=1_1615517185.237222_418.3 target=1_1615516657.657755_238.1 2021-03-12T02:47:41.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:41.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:41.662 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:41.704 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:41.705 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:41.920 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9936002905845503227},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516457.862054_318.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:17.862054Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.706047_386.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.706047Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517043.271340_351.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:44:03.271340Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.657755_238.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.657755Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:47:41.921 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1615517185.237222_553.2 target=1_1615516457.862054_318.3, shard=1 master=1_1615517185.237222_550.3 target=1_1615516820.706047_386.1, shard=2 master=1_1615517185.237222_405.3 target=1_1615517043.271340_351.1, shard=3 master=1_1615517185.237222_418.3 target=1_1615516657.657755_238.1 2021-03-12T02:47:46.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:46.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:46.924 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:46.966 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:46.966 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:47.188 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9936002905845503227},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615516457.862054_318.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:34:17.862054Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615516820.706047_386.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:40:20.706047Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517043.271340_351.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:44:03.271340Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615516657.657755_238.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:37:37.657755Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:47:47.188 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1615517185.237222_553.2 target=1_1615516457.862054_318.3, shard=1 master=1_1615517185.237222_550.3 target=1_1615516820.706047_386.1, shard=2 master=1_1615517185.237222_405.3 target=1_1615517043.271340_351.1, shard=3 master=1_1615517185.237222_418.3 target=1_1615516657.657755_238.1 2021-03-12T02:47:52.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:52.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:52.191 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:52.232 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:52.233 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:52.440 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9936002905845503227},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517185.237222_553.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:25.237222Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517185.237222_550.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:25.237222Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517185.237222_405.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:25.237222Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517185.237222_418.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:25.237222Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:47:52.440 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-12T02:47:52.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:52.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:52.442 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:52.481 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:52.482 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:52.698 INFO:teuthology.orchestra.run.smithi114.stdout:[{"marker":"1_1615517185.237222_553.2","last_update":"2021-03-12T02:46:25.237222Z"},{"marker":"1_1615517185.237222_550.3","last_update":"2021-03-12T02:46:25.237222Z"},{"marker":"1_1615517185.237222_405.3","last_update":"2021-03-12T02:46:25.237222Z"},{"marker":"1_1615517185.237222_418.3","last_update":"2021-03-12T02:46:25.237222Z"}] 2021-03-12T02:47:52.699 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-12T02:47:52.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:52.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:52.700 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:52.743 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:52.744 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:52.953 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3923957601198887203},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517185.237222_553.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:25.237222Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517185.237222_550.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:25.237222Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517185.237222_405.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:25.237222Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517185.237222_418.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:25.237222Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:47:52.954 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-12T02:47:52.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:52.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:52.955 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:53.036 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:53.037 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:53.253 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615517179.574204_496.2","last_update":"2021-03-12T02:46:19.574204Z"},{"marker":"1_1615517179.574204_459.3","last_update":"2021-03-12T02:46:19.574204Z"},{"marker":"1_1615517179.574204_341.3","last_update":"2021-03-12T02:46:19.574204Z"},{"marker":"1_1615517179.574204_330.3","last_update":"2021-03-12T02:46:19.574204Z"}] 2021-03-12T02:47:53.254 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-12T02:47:53.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:53.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:53.255 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:53.336 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:53.336 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:53.570 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12616054323329998860},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517179.574204_496.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:19.574204Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517179.574204_459.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:19.574204Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517179.574204_341.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:19.574204Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517179.574204_330.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:19.574204Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:47:53.570 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-12T02:47:53.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:53.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:53.572 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:53.651 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:53.651 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:53.883 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615517179.574204_496.2","last_update":"2021-03-12T02:46:19.574204Z"},{"marker":"1_1615517179.574204_459.3","last_update":"2021-03-12T02:46:19.574204Z"},{"marker":"1_1615517179.574204_341.3","last_update":"2021-03-12T02:46:19.574204Z"},{"marker":"1_1615517179.574204_330.3","last_update":"2021-03-12T02:46:19.574204Z"}] 2021-03-12T02:47:53.883 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-12T02:47:53.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:53.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:53.885 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:53.926 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:53.927 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:54.143 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10754379456686351807},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517179.574204_496.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:19.574204Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517179.574204_459.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:19.574204Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517179.574204_341.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:19.574204Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517179.574204_330.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:19.574204Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:47:54.144 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-12T02:47:54.145 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... ok 2021-03-12T02:47:54.146 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-59 2021-03-12T02:47:54.195 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-60 2021-03-12T02:47:54.272 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-61 2021-03-12T02:47:54.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:54.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:54.944 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:54.987 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:54.988 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:55.204 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.59.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.58.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.64.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:47:55.205 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-59 2021-03-12T02:47:55.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:55.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:47:55.206 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:47:55.256 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:47:55.256 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:47:55.445 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"full-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.62.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:47:55.446 WARNING:rgw_multi.tests:bucket bbucnh-59 zone test-zone2 behind zone test-zone1: shard=2 master=00000000001.59.5 target=, shard=4 master=00000000001.58.5 target=, shard=7 master=00000000001.64.5 target= 2021-03-12T02:48:00.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:00.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:00.450 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:00.493 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:00.493 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:00.729 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:00.730 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-59 2021-03-12T02:48:00.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:00.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:00.732 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:00.772 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:00.772 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:01.007 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.59.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.58.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.64.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:01.007 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-59 2021-03-12T02:48:01.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:01.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:01.009 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:01.090 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:01.091 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:01.311 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:01.312 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-59 2021-03-12T02:48:01.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:01.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:01.313 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:01.391 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:01.392 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:01.630 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.59.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.58.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.64.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:01.630 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-59 2021-03-12T02:48:01.631 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:01.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:01.632 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:01.676 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:01.939 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.62.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.59.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.58.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.64.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:01.940 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-59 2021-03-12T02:48:01.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:01.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:01.941 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:02.022 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:02.022 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:02.234 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.62.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.43.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.64.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:02.235 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-59 2021-03-12T02:48:02.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:02.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:02.236 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:02.282 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:02.282 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:02.532 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:02.532 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-59 2021-03-12T02:48:02.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:02.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:02.534 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:02.574 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:02.574 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:02.792 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.62.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.43.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.64.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:02.793 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-59 2021-03-12T02:48:02.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:02.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:02.794 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:02.839 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:02.839 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:03.074 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:03.075 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-59 2021-03-12T02:48:03.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:03.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:03.076 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:03.158 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:03.158 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:03.377 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.74.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.53.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.47.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.76.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:03.379 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-59 2021-03-12T02:48:03.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:03.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:03.383 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:03.476 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:03.476 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:03.732 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:03.732 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-59 2021-03-12T02:48:03.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:03.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:03.734 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:03.808 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:03.809 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:04.029 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.74.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.53.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.47.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.76.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:04.030 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-59 2021-03-12T02:48:04.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:04.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:04.032 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:04.076 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:04.076 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:04.314 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:04.315 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-59 2021-03-12T02:48:04.316 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-59 zones={test-zone2, test-zone1} 2021-03-12T02:48:05.571 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:48:05.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:05.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:05.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:05.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:05.580 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:48:05.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:05.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:05.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:05.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:05.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:05.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:05.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:05.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:05.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:06.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:06.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:06.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:06.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:06.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:06.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:06.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:06.756 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-59 zones={test-zone2, test-zone1} 2021-03-12T02:48:06.757 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-59 zones={test-zone3, test-zone1} 2021-03-12T02:48:06.929 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:48:06.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:06.936 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:06.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:06.937 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:06.938 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:48:06.943 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:06.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:06.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:06.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:06.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:07.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:07.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:07.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:07.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:07.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:07.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:07.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:07.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:07.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:07.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:07.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:08.103 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-59 zones={test-zone3, test-zone1} 2021-03-12T02:48:08.104 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-59 zones={test-zone3, test-zone2} 2021-03-12T02:48:08.449 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:48:08.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:08.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:08.465 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:08.466 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:08.466 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:48:08.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:08.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:08.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:08.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:08.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:08.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:08.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:08.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:08.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:08.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:09.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:09.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:09.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:09.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:09.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:09.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:09.609 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-59 zones={test-zone3, test-zone2} 2021-03-12T02:48:09.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:09.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:09.611 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:09.654 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:09.655 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:09.886 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.88.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.60.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.52.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.44.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:09.887 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-60 2021-03-12T02:48:09.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:09.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:09.889 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:09.934 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:09.935 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:10.159 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.88.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:10.160 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-60 2021-03-12T02:48:10.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:10.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:10.161 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:10.201 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:10.201 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:10.426 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.88.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.60.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.52.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.44.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:10.427 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-60 2021-03-12T02:48:10.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:10.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:10.428 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:10.510 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:10.510 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:10.761 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.88.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:10.762 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-60 2021-03-12T02:48:10.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:10.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:10.763 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:10.844 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:10.844 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:11.064 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.88.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.60.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.52.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.44.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:11.065 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-60 2021-03-12T02:48:11.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:11.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:11.067 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:11.113 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:11.114 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:11.379 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.88.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.60.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.52.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.44.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:11.380 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-60 2021-03-12T02:48:11.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:11.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:11.381 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:11.423 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:11.423 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:11.634 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000001.105.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.38.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.47.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.43.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:11.634 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-60 2021-03-12T02:48:11.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:11.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:11.636 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:11.684 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:11.929 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.105.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:11.929 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-60 2021-03-12T02:48:11.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:11.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:11.931 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:11.973 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:11.973 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:12.188 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000001.105.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.38.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.47.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.43.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:12.189 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-60 2021-03-12T02:48:12.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:12.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:12.190 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:12.234 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:12.235 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:12.469 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.105.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:12.470 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-60 2021-03-12T02:48:12.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:12.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:12.471 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:12.553 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:12.553 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:12.765 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.100.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.63.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.44.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:12.765 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-60 2021-03-12T02:48:12.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:12.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:12.767 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:12.848 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:12.848 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:13.079 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.100.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:13.079 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-60 2021-03-12T02:48:13.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:13.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:13.081 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:13.162 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:13.163 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:13.366 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.100.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.63.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.44.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:13.367 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-60 2021-03-12T02:48:13.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:13.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:13.368 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:13.412 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:13.412 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:13.615 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.100.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:13.617 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-60 2021-03-12T02:48:13.618 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-60 zones={test-zone2, test-zone1} 2021-03-12T02:48:14.531 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:48:14.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:14.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:14.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:14.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:14.539 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:48:14.547 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:14.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:14.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:14.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:14.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:14.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:14.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:14.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:14.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:15.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:15.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:15.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:15.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:15.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:15.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:15.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:15.663 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-60 zones={test-zone2, test-zone1} 2021-03-12T02:48:15.664 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-60 zones={test-zone3, test-zone1} 2021-03-12T02:48:15.682 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:48:15.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:15.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:15.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:15.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:15.693 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:48:15.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:15.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:15.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:15.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:15.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:15.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:15.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:15.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:16.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:16.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:16.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:16.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:16.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:16.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:16.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:16.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:16.902 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-60 zones={test-zone3, test-zone1} 2021-03-12T02:48:16.902 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-60 zones={test-zone3, test-zone2} 2021-03-12T02:48:17.873 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:48:17.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:17.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:17.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:17.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:17.884 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:48:17.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:17.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:17.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:17.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:17.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:17.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:18.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:18.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:18.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:18.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:18.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:18.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:18.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:18.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:18.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:18.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:19.048 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-60 zones={test-zone3, test-zone2} 2021-03-12T02:48:19.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:19.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:19.050 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:19.092 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:19.093 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:19.304 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.52.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.41.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.32.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.39.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:19.304 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-61 2021-03-12T02:48:19.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:19.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:19.306 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:19.351 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:19.351 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:19.573 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.32.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:19.574 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-61 2021-03-12T02:48:19.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:19.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:19.576 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:19.615 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:19.616 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:19.842 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.52.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.41.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.32.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.39.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:19.842 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-61 2021-03-12T02:48:19.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:19.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:19.844 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:19.934 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:19.935 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:20.158 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.32.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:20.158 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-61 2021-03-12T02:48:20.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:20.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:20.160 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:20.201 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:20.201 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:20.423 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.52.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.41.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.32.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.39.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:20.423 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-61 2021-03-12T02:48:20.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:20.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:20.425 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:20.469 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:20.470 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:20.730 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.52.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.41.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.32.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.39.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:20.731 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-61 2021-03-12T02:48:20.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:20.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:20.733 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:20.817 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:20.818 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:21.017 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000001.68.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.25.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.40.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:21.018 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-61 2021-03-12T02:48:21.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:21.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:21.020 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:21.064 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:21.065 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:21.311 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.25.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:21.312 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-61 2021-03-12T02:48:21.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:21.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:21.313 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:21.355 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:21.356 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:21.561 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000001.68.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.25.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.40.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:21.561 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-61 2021-03-12T02:48:21.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:21.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:21.563 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:21.611 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:21.612 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:21.838 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.25.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:21.839 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-61 2021-03-12T02:48:21.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:21.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:21.840 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:21.922 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:21.923 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:22.139 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.52.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.47.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.32.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.49.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:22.139 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-61 2021-03-12T02:48:22.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:22.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:22.141 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:22.224 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:22.224 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:22.456 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.32.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:22.456 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-61 2021-03-12T02:48:22.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:22.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:22.458 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:22.537 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:22.537 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:22.750 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.52.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.47.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.32.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.49.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:22.750 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-61 2021-03-12T02:48:22.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:22.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:22.752 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:22.796 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:22.797 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:23.018 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.32.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:23.019 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-61 2021-03-12T02:48:23.019 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-61 zones={test-zone2, test-zone1} 2021-03-12T02:48:24.063 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:48:24.071 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:24.072 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:24.073 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:24.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:24.075 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:48:24.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:24.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:24.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:24.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:24.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:24.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:24.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:24.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:24.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:24.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:24.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:24.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:24.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:24.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:25.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:25.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:25.265 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-61 zones={test-zone2, test-zone1} 2021-03-12T02:48:25.266 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-61 zones={test-zone3, test-zone1} 2021-03-12T02:48:25.398 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:48:25.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:25.404 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:25.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:25.405 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:25.406 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:48:25.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:25.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:25.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:25.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:25.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:25.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:25.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:25.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:25.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:25.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:25.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:26.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:26.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:26.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:26.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:26.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:26.568 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-61 zones={test-zone3, test-zone1} 2021-03-12T02:48:26.568 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-61 zones={test-zone3, test-zone2} 2021-03-12T02:48:27.383 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:48:27.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:27.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:27.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:27.394 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:27.394 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:48:27.400 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:27.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:27.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:27.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:27.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:27.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:27.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:27.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:27.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:27.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:27.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:28.042 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:28.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:28.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:28.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:28.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:28.556 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-61 zones={test-zone3, test-zone2} 2021-03-12T02:48:28.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'bbucnh-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:48:28.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'bbucnh-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:48:28.558 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket bbucnh-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:48:28.600 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:28.601 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:28.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'bbucnh-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:48:28.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'bbucnh-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:48:28.872 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket bbucnh-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:48:28.954 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:28.954 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:29.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'bbucnh-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:48:29.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'bbucnh-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:48:29.233 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket bbucnh-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:48:29.316 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:29.316 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:29.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'bbucnh-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:48:29.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'bbucnh-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:48:29.597 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket bbucnh-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:48:29.677 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:29.678 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:29.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'bbucnh-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:48:29.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'bbucnh-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:48:29.959 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket bbucnh-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:48:30.040 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:30.041 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:30.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'bbucnh-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:48:30.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'bbucnh-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:48:30.322 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket bbucnh-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:48:30.410 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:30.411 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:33.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:33.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:33.300 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:33.341 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:33.341 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:33.527 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.78.5"},{"key":1,"val":"00000000004.73.1"},{"key":2,"val":"00000000006.69.5"},{"key":3,"val":"00000000004.42.1"},{"key":4,"val":"00000000006.74.5"},{"key":5,"val":"00000000004.75.1"},{"key":6,"val":"00000000004.48.1"},{"key":7,"val":"00000000006.80.5"},{"key":8,"val":"00000000004.68.1"},{"key":9,"val":"00000000004.36.1"},{"key":10,"val":"00000000004.54.1"}]} 2021-03-12T02:48:33.528 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-59 2021-03-12T02:48:33.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:33.529 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:33.530 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:33.577 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:33.578 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:33.789 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.78.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.69.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.74.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1"}}},{"key":7,"val":{"status":"full-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.80.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}}] 2021-03-12T02:48:33.790 WARNING:rgw_multi.tests:bucket bbucnh-59 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.73.1 target=, shard=5 master=00000000004.75.1 target=, shard=9 master=00000000004.36.1 target= 2021-03-12T02:48:38.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:38.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:38.793 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:38.836 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:38.836 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:39.066 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.78.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.69.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.74.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.80.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}}] 2021-03-12T02:48:39.067 WARNING:rgw_multi.tests:bucket bbucnh-59 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.73.1 target=, shard=5 master=00000000004.75.1 target=, shard=9 master=00000000004.36.1 target= 2021-03-12T02:48:44.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:44.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:44.069 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:44.115 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:44.115 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:44.344 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.78.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.69.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.74.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.80.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}}] 2021-03-12T02:48:44.345 WARNING:rgw_multi.tests:bucket bbucnh-59 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.73.1 target=, shard=5 master=00000000004.75.1 target=, shard=9 master=00000000004.36.1 target= 2021-03-12T02:48:49.347 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:49.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:49.348 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:49.392 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:49.392 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:49.621 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.78.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.69.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.74.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.80.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.36.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}}] 2021-03-12T02:48:49.622 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-59 2021-03-12T02:48:49.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:49.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:49.623 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:49.665 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:49.666 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:49.885 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.78.5"},{"key":1,"val":"00000000004.73.1"},{"key":2,"val":"00000000006.69.5"},{"key":3,"val":"00000000004.42.1"},{"key":4,"val":"00000000006.74.5"},{"key":5,"val":"00000000004.75.1"},{"key":6,"val":"00000000004.48.1"},{"key":7,"val":"00000000006.80.5"},{"key":8,"val":"00000000004.68.1"},{"key":9,"val":"00000000004.36.1"},{"key":10,"val":"00000000004.54.1"}]} 2021-03-12T02:48:49.885 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-59 2021-03-12T02:48:49.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:49.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:49.887 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:49.972 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:49.973 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:50.162 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.78.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.69.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.74.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.80.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.36.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}}] 2021-03-12T02:48:50.163 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-59 2021-03-12T02:48:50.164 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:50.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:50.165 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:50.244 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:50.244 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:50.456 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.78.5"},{"key":1,"val":"00000000004.73.1"},{"key":2,"val":"00000000006.69.5"},{"key":3,"val":"00000000004.42.1"},{"key":4,"val":"00000000006.74.5"},{"key":5,"val":"00000000004.75.1"},{"key":6,"val":"00000000004.48.1"},{"key":7,"val":"00000000006.80.5"},{"key":8,"val":"00000000004.68.1"},{"key":9,"val":"00000000004.36.1"},{"key":10,"val":"00000000004.54.1"}]} 2021-03-12T02:48:50.457 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-59 2021-03-12T02:48:50.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:50.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:50.459 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:50.502 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:50.502 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:50.768 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.78.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.69.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.74.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.80.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.36.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}}] 2021-03-12T02:48:50.769 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-59 2021-03-12T02:48:50.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:50.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:50.770 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:50.858 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:50.858 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:51.076 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000002.72.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.68.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.53.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.74.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:51.076 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-59 2021-03-12T02:48:51.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:51.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:51.078 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:51.121 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:51.122 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:51.341 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.72.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.68.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.53.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.74.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:51.341 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-59 2021-03-12T02:48:51.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:51.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:51.343 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:51.384 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:51.385 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:51.598 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000002.72.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.68.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.53.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.74.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:51.599 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-59 2021-03-12T02:48:51.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:51.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:51.601 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:51.645 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:51.645 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:51.890 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.72.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.68.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.53.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.74.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:51.890 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-59 2021-03-12T02:48:51.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:51.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:51.892 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:51.933 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:51.934 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:52.147 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000002.78.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.55.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.51.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.80.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:52.147 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-59 2021-03-12T02:48:52.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:52.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:52.149 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:52.230 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:52.231 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:52.459 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.78.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.55.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.51.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.80.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:52.460 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-59 2021-03-12T02:48:52.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:52.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:52.461 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:52.540 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:52.540 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:52.763 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000002.78.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.55.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.51.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.80.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:48:52.763 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-59 2021-03-12T02:48:52.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:52.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:48:52.765 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:48:52.810 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:48:52.810 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:48:53.037 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.78.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.55.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.51.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.80.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:48:53.038 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-59 2021-03-12T02:48:53.039 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-59 zones={test-zone2, test-zone1} 2021-03-12T02:48:54.820 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:48:54.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:54.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:54.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:54.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:54.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:48:54.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:48:54.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:48:54.831 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:48:54.831 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:48:54.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:54.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:54.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:54.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:54.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:48:54.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:48:54.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:48:54.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:48:54.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:54.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:55.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:55.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:55.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:55.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:55.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:55.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:55.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:55.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:55.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:55.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:56.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:48:56.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:48:56.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:48:56.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:48:56.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:48:56.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:48:56.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:48:56.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:48:56.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:48:56.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:48:56.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:48:57.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:48:57.119 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-59 zones={test-zone2, test-zone1} 2021-03-12T02:48:57.120 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-59 zones={test-zone3, test-zone1} 2021-03-12T02:48:57.886 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:48:57.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:57.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:57.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:57.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:57.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:48:57.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:48:57.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:48:57.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:48:57.897 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:48:57.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:48:57.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:48:57.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:48:57.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:48:57.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:48:57.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:48:57.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:48:57.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:48:57.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:57.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:58.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:48:58.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:58.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:58.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:48:58.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:58.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:58.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:48:58.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:58.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:58.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:48:59.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:48:59.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:48:59.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:48:59.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:48:59.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:48:59.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:48:59.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:48:59.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:48:59.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:48:59.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:48:59.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:00.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:00.174 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-59 zones={test-zone3, test-zone1} 2021-03-12T02:49:00.174 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-59 zones={test-zone3, test-zone2} 2021-03-12T02:49:00.968 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:49:00.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:00.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:00.980 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:00.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:00.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:49:00.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:49:00.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:49:00.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:49:00.983 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:49:00.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:00.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:00.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:00.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:00.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:49:00.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:49:00.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:49:00.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:49:01.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:01.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:01.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:01.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:01.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:01.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:01.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:01.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:01.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:01.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:01.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:01.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:02.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:02.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:02.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:02.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:02.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:02.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:02.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:02.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:02.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:02.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:02.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:03.058 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:03.165 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-59 zones={test-zone3, test-zone2} 2021-03-12T02:49:03.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:03.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:03.167 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:03.211 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:03.212 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:03.440 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.94.5"},{"key":1,"val":"00000000004.67.1"},{"key":2,"val":"00000000006.76.5"},{"key":3,"val":"00000000004.35.1"},{"key":4,"val":"00000000006.66.5"},{"key":5,"val":"00000000004.58.1"},{"key":6,"val":"00000000004.68.1"},{"key":7,"val":"00000000006.54.5"},{"key":8,"val":"00000000004.60.1"},{"key":9,"val":"00000000004.53.1"},{"key":10,"val":"00000000004.59.1"}]} 2021-03-12T02:49:03.440 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-60 2021-03-12T02:49:03.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:03.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:03.442 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:03.487 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:03.488 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:03.716 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.94.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.76.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.66.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.54.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1"}}}] 2021-03-12T02:49:03.716 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-60 2021-03-12T02:49:03.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:03.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:03.718 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:03.760 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:03.761 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:04.002 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.94.5"},{"key":1,"val":"00000000004.67.1"},{"key":2,"val":"00000000006.76.5"},{"key":3,"val":"00000000004.35.1"},{"key":4,"val":"00000000006.66.5"},{"key":5,"val":"00000000004.58.1"},{"key":6,"val":"00000000004.68.1"},{"key":7,"val":"00000000006.54.5"},{"key":8,"val":"00000000004.60.1"},{"key":9,"val":"00000000004.53.1"},{"key":10,"val":"00000000004.59.1"}]} 2021-03-12T02:49:04.003 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-60 2021-03-12T02:49:04.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:04.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:04.004 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:04.086 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:04.086 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:04.327 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.94.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.76.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.66.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.54.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1"}}}] 2021-03-12T02:49:04.328 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-60 2021-03-12T02:49:04.328 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:04.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:04.329 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:04.409 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:04.410 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:04.636 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.94.5"},{"key":1,"val":"00000000004.67.1"},{"key":2,"val":"00000000006.76.5"},{"key":3,"val":"00000000004.35.1"},{"key":4,"val":"00000000006.66.5"},{"key":5,"val":"00000000004.58.1"},{"key":6,"val":"00000000004.68.1"},{"key":7,"val":"00000000006.54.5"},{"key":8,"val":"00000000004.60.1"},{"key":9,"val":"00000000004.53.1"},{"key":10,"val":"00000000004.59.1"}]} 2021-03-12T02:49:04.637 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-60 2021-03-12T02:49:04.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:04.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:04.638 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:04.683 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:04.683 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:04.948 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.94.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.76.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.66.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.54.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1"}}}] 2021-03-12T02:49:04.948 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-60 2021-03-12T02:49:04.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:04.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:04.950 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:05.032 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:05.032 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:05.255 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000002.108.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.52.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.47.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:05.256 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-60 2021-03-12T02:49:05.257 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:05.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:05.258 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:05.300 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:05.301 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:05.542 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.108.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.45.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.52.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.47.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:05.543 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-60 2021-03-12T02:49:05.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:05.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:05.546 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:05.590 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:05.591 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:05.802 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000002.108.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.52.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.47.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:05.803 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-60 2021-03-12T02:49:05.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:05.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:05.804 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:05.848 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:05.849 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:06.092 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.108.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.45.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.52.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.47.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:06.092 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-60 2021-03-12T02:49:06.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:06.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:06.094 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:06.171 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:06.172 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:06.391 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000002.102.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.53.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.65.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.46.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:06.391 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-60 2021-03-12T02:49:06.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:06.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:06.393 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:06.478 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:06.478 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:06.730 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.102.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.53.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.65.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.46.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:06.731 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-60 2021-03-12T02:49:06.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:06.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:06.733 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:06.815 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:06.816 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:07.034 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000002.102.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.53.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.65.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.46.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:07.035 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-60 2021-03-12T02:49:07.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:07.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:07.036 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:07.081 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:07.081 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:07.307 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.102.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.53.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.65.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.46.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:07.308 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-60 2021-03-12T02:49:07.308 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-60 zones={test-zone2, test-zone1} 2021-03-12T02:49:09.018 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:49:09.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:09.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:09.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:09.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:09.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:49:09.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:49:09.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:49:09.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:49:09.028 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:49:09.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:09.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:09.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:09.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:09.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:49:09.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:49:09.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:49:09.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:49:09.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:09.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:09.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:09.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:09.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:09.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:09.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:09.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:09.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:09.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:09.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:10.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:10.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:10.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:10.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:10.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:10.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:10.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:10.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:10.750 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:10.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:10.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:11.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:11.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:11.242 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-60 zones={test-zone2, test-zone1} 2021-03-12T02:49:11.243 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-60 zones={test-zone3, test-zone1} 2021-03-12T02:49:12.030 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:49:12.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:12.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:12.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:12.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:12.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:49:12.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:49:12.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:49:12.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:49:12.039 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:49:12.045 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:12.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:12.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:12.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:12.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:49:12.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:49:12.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:49:12.048 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:49:12.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:12.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:12.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:12.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:12.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:12.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:12.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:12.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:12.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:12.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:12.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:13.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:13.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:13.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:13.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:13.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:13.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:13.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:13.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:13.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:13.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:14.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:14.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:14.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:14.310 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-60 zones={test-zone3, test-zone1} 2021-03-12T02:49:14.311 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-60 zones={test-zone3, test-zone2} 2021-03-12T02:49:15.266 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:49:15.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:15.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:15.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:15.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:15.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:49:15.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:49:15.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:49:15.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:49:15.277 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:49:15.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:15.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:15.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:15.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:15.285 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:49:15.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:49:15.286 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:49:15.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:49:15.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:15.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:15.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:15.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:15.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:15.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:15.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:15.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:15.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:16.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:16.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:16.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:16.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:16.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:16.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:16.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:16.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:16.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:16.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:16.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:17.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:17.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:17.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:17.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:17.464 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-60 zones={test-zone3, test-zone2} 2021-03-12T02:49:17.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:17.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:17.466 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:17.508 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:17.508 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:17.728 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.58.5"},{"key":1,"val":"00000000004.75.1"},{"key":2,"val":"00000000006.55.5"},{"key":3,"val":"00000000004.52.1"},{"key":4,"val":"00000000006.38.5"},{"key":5,"val":"00000000004.53.1"},{"key":6,"val":"00000000004.74.1"},{"key":7,"val":"00000000006.45.5"},{"key":8,"val":"00000000004.64.1"},{"key":9,"val":"00000000004.62.1"},{"key":10,"val":"00000000004.77.1"}]} 2021-03-12T02:49:17.729 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-61 2021-03-12T02:49:17.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:17.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:17.730 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:17.775 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:17.776 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:18.028 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.58.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.55.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.38.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.45.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-12T02:49:18.029 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-61 2021-03-12T02:49:18.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:18.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:18.030 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:18.071 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:18.072 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:18.295 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.58.5"},{"key":1,"val":"00000000004.75.1"},{"key":2,"val":"00000000006.55.5"},{"key":3,"val":"00000000004.52.1"},{"key":4,"val":"00000000006.38.5"},{"key":5,"val":"00000000004.53.1"},{"key":6,"val":"00000000004.74.1"},{"key":7,"val":"00000000006.45.5"},{"key":8,"val":"00000000004.64.1"},{"key":9,"val":"00000000004.62.1"},{"key":10,"val":"00000000004.77.1"}]} 2021-03-12T02:49:18.295 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-61 2021-03-12T02:49:18.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:18.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:18.297 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:18.378 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:18.379 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:18.614 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.58.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.55.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.38.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.45.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-12T02:49:18.614 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-61 2021-03-12T02:49:18.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:18.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:18.616 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:18.700 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:18.700 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:18.927 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.58.5"},{"key":1,"val":"00000000004.75.1"},{"key":2,"val":"00000000006.55.5"},{"key":3,"val":"00000000004.52.1"},{"key":4,"val":"00000000006.38.5"},{"key":5,"val":"00000000004.53.1"},{"key":6,"val":"00000000004.74.1"},{"key":7,"val":"00000000006.45.5"},{"key":8,"val":"00000000004.64.1"},{"key":9,"val":"00000000004.62.1"},{"key":10,"val":"00000000004.77.1"}]} 2021-03-12T02:49:18.927 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-61 2021-03-12T02:49:18.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:18.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:18.929 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:18.972 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:18.973 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:19.234 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.58.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.55.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.38.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.45.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-12T02:49:19.235 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-61 2021-03-12T02:49:19.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:19.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:19.237 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:19.316 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:19.316 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:19.525 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000002.72.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.53.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.29.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.44.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:19.526 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-61 2021-03-12T02:49:19.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:19.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:19.527 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:19.572 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:19.572 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:19.817 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.72.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.53.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.29.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.44.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:19.817 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-61 2021-03-12T02:49:19.818 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:19.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:19.819 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:19.861 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:19.861 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:20.061 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000002.72.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.53.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.29.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.44.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:20.061 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-61 2021-03-12T02:49:20.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:20.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:20.063 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:20.106 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:20.106 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:20.341 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.72.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.53.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.29.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.44.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:20.341 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-61 2021-03-12T02:49:20.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:20.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:20.343 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:20.423 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:20.423 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:20.643 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000002.54.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.34.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.51.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:20.644 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-61 2021-03-12T02:49:20.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:20.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:20.645 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:20.731 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:20.732 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:20.984 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.54.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.49.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.34.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.51.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:20.985 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-61 2021-03-12T02:49:20.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:20.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:20.990 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:21.068 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:21.068 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:21.287 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000002.54.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.34.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.51.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:21.288 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-61 2021-03-12T02:49:21.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:21.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:21.290 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:21.333 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:21.333 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:21.568 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.54.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.49.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.34.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.51.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:21.568 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-61 2021-03-12T02:49:21.569 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-61 zones={test-zone2, test-zone1} 2021-03-12T02:49:22.577 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:49:22.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:22.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:22.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:22.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:22.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:49:22.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:49:22.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:49:22.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:49:22.589 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:49:22.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:22.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:22.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:22.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:22.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:49:22.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:49:22.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:49:22.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:49:22.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:22.674 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:22.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:22.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:22.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:23.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:23.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:23.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:23.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:23.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:23.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:23.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:23.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:23.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:23.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:24.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:24.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:24.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:24.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:24.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:24.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:24.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:24.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:24.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:24.851 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-61 zones={test-zone2, test-zone1} 2021-03-12T02:49:24.852 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-61 zones={test-zone3, test-zone1} 2021-03-12T02:49:25.160 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:49:25.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:25.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:25.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:25.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:25.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:49:25.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:49:25.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:49:25.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:49:25.170 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:49:25.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:25.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:25.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:25.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:25.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:49:25.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:49:25.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:49:25.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:49:25.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:25.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:25.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:25.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:25.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:25.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:25.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:25.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:25.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:26.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:26.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:26.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:26.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:26.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:26.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:26.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:26.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:26.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:26.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:26.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:27.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:27.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:27.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:27.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:27.424 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-61 zones={test-zone3, test-zone1} 2021-03-12T02:49:27.425 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-61 zones={test-zone3, test-zone2} 2021-03-12T02:49:28.199 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:49:28.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:28.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:28.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:28.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:28.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:49:28.213 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:49:28.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:49:28.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:49:28.214 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:49:28.222 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:28.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:28.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:28.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:28.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:49:28.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:49:28.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:49:28.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:49:28.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:28.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:28.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:28.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:28.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:28.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:28.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:28.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:28.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:29.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:29.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:29.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:29.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:29.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:29.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:49:29.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:29.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:29.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:49:29.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:29.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:30.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:49:30.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:30.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:30.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:49:30.486 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-61 zones={test-zone3, test-zone2} 2021-03-12T02:49:30.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:30.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:30.487 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:30.531 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:30.531 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:30.754 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615517315.119479_543.1","last_update":"2021-03-12T02:48:35.119479Z"},{"marker":"1_1615517314.039345_684.1","last_update":"2021-03-12T02:48:34.039345Z"},{"marker":"1_1615517310.644969_617.3","last_update":"2021-03-12T02:48:30.644969Z"},{"marker":"1_1615517314.038114_542.1","last_update":"2021-03-12T02:48:34.038114Z"}] 2021-03-12T02:49:30.755 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-12T02:49:30.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:30.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:30.756 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:30.799 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:30.799 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:31.011 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8376578737807985417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517315.119479_543.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:35.119479Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517314.039345_684.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:34.039345Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517310.644969_617.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:30.644969Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517314.038114_542.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:34.038114Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:49:31.012 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-12T02:49:31.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:31.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:31.013 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:31.053 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:31.054 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:31.294 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615517315.119479_543.1","last_update":"2021-03-12T02:48:35.119479Z"},{"marker":"1_1615517314.039345_684.1","last_update":"2021-03-12T02:48:34.039345Z"},{"marker":"1_1615517310.644969_617.3","last_update":"2021-03-12T02:48:30.644969Z"},{"marker":"1_1615517314.038114_542.1","last_update":"2021-03-12T02:48:34.038114Z"}] 2021-03-12T02:49:31.294 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-12T02:49:31.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:31.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:31.296 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:31.376 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:31.376 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:31.585 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1279455150295362088},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517315.119479_543.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:35.119479Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517314.039345_684.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:34.039345Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517310.644969_617.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:30.644969Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517314.038114_542.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:34.038114Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:49:31.585 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-12T02:49:31.586 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:31.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:31.587 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:31.667 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:31.668 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:31.888 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615517315.119479_543.1","last_update":"2021-03-12T02:48:35.119479Z"},{"marker":"1_1615517314.039345_684.1","last_update":"2021-03-12T02:48:34.039345Z"},{"marker":"1_1615517310.644969_617.3","last_update":"2021-03-12T02:48:30.644969Z"},{"marker":"1_1615517314.038114_542.1","last_update":"2021-03-12T02:48:34.038114Z"}] 2021-03-12T02:49:31.889 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-12T02:49:31.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:31.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:31.890 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:31.933 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:31.933 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:32.195 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11911835907256974194},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517315.119479_543.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:35.119479Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517314.039345_684.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:34.039345Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517310.644969_617.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:30.644969Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517314.038114_542.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:34.038114Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:49:32.196 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-12T02:49:32.196 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:32.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:32.197 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:32.279 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:32.280 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:32.481 INFO:teuthology.orchestra.run.smithi114.stdout:[{"marker":"1_1615517315.077560_570.1","last_update":"2021-03-12T02:48:35.077560Z"},{"marker":"1_1615517314.075146_594.1","last_update":"2021-03-12T02:48:34.075146Z"},{"marker":"1_1615517185.237222_405.3","last_update":"2021-03-12T02:46:25.237222Z"},{"marker":"1_1615517314.047851_467.1","last_update":"2021-03-12T02:48:34.047851Z"}] 2021-03-12T02:49:32.482 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-12T02:49:32.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:32.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:32.484 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:32.525 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:32.525 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:32.763 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9936002905845503227},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517315.077560_570.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:35.077560Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517314.075146_594.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:34.075146Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517185.237222_405.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:25.237222Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517314.047851_467.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:34.047851Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:49:32.763 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-12T02:49:32.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:32.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:32.765 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:32.806 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:32.807 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:33.017 INFO:teuthology.orchestra.run.smithi114.stdout:[{"marker":"1_1615517315.077560_570.1","last_update":"2021-03-12T02:48:35.077560Z"},{"marker":"1_1615517314.075146_594.1","last_update":"2021-03-12T02:48:34.075146Z"},{"marker":"1_1615517185.237222_405.3","last_update":"2021-03-12T02:46:25.237222Z"},{"marker":"1_1615517314.047851_467.1","last_update":"2021-03-12T02:48:34.047851Z"}] 2021-03-12T02:49:33.018 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-12T02:49:33.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:33.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:33.019 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:33.060 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:33.061 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:33.277 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3923957601198887203},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517315.077560_570.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:35.077560Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517314.075146_594.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:34.075146Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517185.237222_405.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:25.237222Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517314.047851_467.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:34.047851Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:49:33.277 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-12T02:49:33.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:33.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:33.279 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:33.358 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:33.358 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:33.572 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615517313.855769_510.1","last_update":"2021-03-12T02:48:33.855769Z"},{"marker":"1_1615517313.386970_501.1","last_update":"2021-03-12T02:48:33.386970Z"},{"marker":"1_1615517179.574204_341.3","last_update":"2021-03-12T02:46:19.574204Z"},{"marker":"1_1615517313.317615_376.1","last_update":"2021-03-12T02:48:33.317615Z"}] 2021-03-12T02:49:33.572 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-12T02:49:33.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:33.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:33.574 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:33.654 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:33.654 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:33.896 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12616054323329998860},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517313.855769_510.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:33.855769Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517313.386970_501.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:33.386970Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517179.574204_341.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:19.574204Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517313.317615_376.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:33.317615Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:49:33.896 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-12T02:49:33.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:33.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:33.898 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:33.976 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:33.976 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:34.208 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615517313.855769_510.1","last_update":"2021-03-12T02:48:33.855769Z"},{"marker":"1_1615517313.386970_501.1","last_update":"2021-03-12T02:48:33.386970Z"},{"marker":"1_1615517179.574204_341.3","last_update":"2021-03-12T02:46:19.574204Z"},{"marker":"1_1615517313.317615_376.1","last_update":"2021-03-12T02:48:33.317615Z"}] 2021-03-12T02:49:34.208 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-12T02:49:34.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:34.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:34.210 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:34.251 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:34.252 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:34.452 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10754379456686351807},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517313.855769_510.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:33.855769Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517313.386970_501.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:33.386970Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517179.574204_341.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:46:19.574204Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517313.317615_376.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:48:33.317615Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:49:34.452 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-12T02:49:34.453 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2021-03-12T02:49:34.454 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-62 2021-03-12T02:49:34.550 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-63 2021-03-12T02:49:35.507 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-64 2021-03-12T02:49:36.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:36.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:36.384 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:36.428 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:36.429 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:36.627 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:49:36.655 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517375.751736_1107.1","last_update":"2021-03-12T02:49:35.751736Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:49:36.656 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:49:36.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:36.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:36.657 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:36.699 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:36.699 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:36.907 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517375.751736_1107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:49:35.751736Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:49:36.908 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:49:36.908 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:49:36.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:36.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:36.910 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:36.948 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:36.948 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:37.453 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517375.751736_1107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:49:35.751736Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:49:37.453 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:49:37.454 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:49:37.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:37.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:37.455 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:37.496 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:37.496 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:37.749 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517375.751736_1107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:49:35.751736Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:49:37.750 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:49:37.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:37.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:37.751 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:37.798 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:37.799 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:37.995 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.50.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.48.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.45.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.70.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:37.996 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-62 2021-03-12T02:49:37.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:37.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:37.999 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:38.041 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:38.259 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.70.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:38.259 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-62 2021-03-12T02:49:38.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:38.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:38.261 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:38.299 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:38.300 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:38.499 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.50.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.48.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.45.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.70.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:38.499 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-62 2021-03-12T02:49:38.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:38.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:38.501 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:38.586 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:38.587 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:38.823 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.70.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:38.824 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-62 2021-03-12T02:49:38.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:38.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:38.826 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:38.911 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:38.912 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:39.109 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.50.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.48.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.45.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.70.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:39.110 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-62 2021-03-12T02:49:39.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:39.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:39.112 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:39.155 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:39.155 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:39.419 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.50.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.48.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.45.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.70.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:39.420 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-62 2021-03-12T02:49:39.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:39.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:39.421 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:39.502 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:39.502 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:39.702 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000001.50.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.44.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.62.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:39.703 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-62 2021-03-12T02:49:39.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:39.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:39.704 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:39.750 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:39.750 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:40.006 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:40.007 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-62 2021-03-12T02:49:40.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:40.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:40.009 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:40.051 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:40.051 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:40.254 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000001.50.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.44.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.62.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:40.255 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-62 2021-03-12T02:49:40.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:40.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:40.256 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:40.303 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:40.303 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:40.558 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:40.607 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-62 2021-03-12T02:49:40.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:40.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:40.609 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:40.652 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:40.652 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:40.884 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.59.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.48.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.76.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:40.884 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-62 2021-03-12T02:49:40.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:40.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:40.886 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:40.966 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:40.966 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:41.208 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:41.209 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-62 2021-03-12T02:49:41.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:41.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:41.211 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:41.526 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.59.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.45.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.48.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.76.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:41.527 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-62 2021-03-12T02:49:41.527 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:41.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:41.528 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:41.572 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:41.573 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:41.787 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:41.788 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-62 2021-03-12T02:49:41.788 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-62 zones={test-zone2, test-zone1} 2021-03-12T02:49:43.173 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:49:43.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:43.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:43.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:43.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:43.181 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:49:43.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:43.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:43.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:43.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:43.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:43.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:43.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:43.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:43.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:43.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:43.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:43.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:43.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:44.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:44.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:44.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:44.278 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-62 zones={test-zone2, test-zone1} 2021-03-12T02:49:44.279 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-62 zones={test-zone3, test-zone1} 2021-03-12T02:49:45.246 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:49:45.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:45.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:45.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:45.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:45.253 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:49:45.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:45.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:45.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:45.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:45.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:45.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:45.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:45.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:45.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:45.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:45.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:45.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:45.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:46.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:46.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:46.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:46.365 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-62 zones={test-zone3, test-zone1} 2021-03-12T02:49:46.365 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-62 zones={test-zone3, test-zone2} 2021-03-12T02:49:47.075 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:49:47.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:47.084 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:47.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:47.085 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:47.086 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:49:47.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:47.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:47.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:47.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:47.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:47.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:47.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:47.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:47.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:47.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:47.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:47.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:47.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:47.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:47.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:48.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:48.176 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-62 zones={test-zone3, test-zone2} 2021-03-12T02:49:48.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:48.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:48.178 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:48.218 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:48.219 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:48.448 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.56.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.47.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.78.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.73.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:48.449 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-63 2021-03-12T02:49:48.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:48.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:48.450 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:48.495 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:48.495 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:48.709 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:48.710 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-63 2021-03-12T02:49:48.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:48.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:48.712 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:48.755 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:48.756 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:48.979 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.56.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.47.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.78.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.73.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:48.980 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-63 2021-03-12T02:49:48.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:48.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:48.982 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:49.065 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:49.065 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:49.301 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:49.302 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-63 2021-03-12T02:49:49.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:49.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:49.304 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:49.342 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:49.342 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:49.564 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.56.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.47.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.78.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.73.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:49.565 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-63 2021-03-12T02:49:49.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:49.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:49.567 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:49.612 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:49.613 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:49.874 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.56.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.47.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.78.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.73.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:49.875 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-63 2021-03-12T02:49:49.875 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:49.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:49.877 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:49.917 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:49.918 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:50.133 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000001.54.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.50.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.95.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.73.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:50.134 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-63 2021-03-12T02:49:50.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:50.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:50.135 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:50.178 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:50.178 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:50.424 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.95.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:50.425 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-63 2021-03-12T02:49:50.425 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:50.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:50.426 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:50.467 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:50.467 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:50.691 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000001.54.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.50.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.95.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.73.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:50.692 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-63 2021-03-12T02:49:50.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:50.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:50.694 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:50.737 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:50.737 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:50.997 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.95.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:50.997 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-63 2021-03-12T02:49:50.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:50.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:50.999 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:51.079 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:51.080 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:51.309 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000002.59.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.47.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.86.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.63.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:51.309 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-63 2021-03-12T02:49:51.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:51.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:51.311 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:51.394 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:51.395 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:51.631 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.59.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.86.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:51.632 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-63 2021-03-12T02:49:51.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:51.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:51.635 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:51.717 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:51.718 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:51.939 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000002.59.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.47.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.86.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.63.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:51.939 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-63 2021-03-12T02:49:51.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:51.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:51.941 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:51.986 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:51.986 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:52.218 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.59.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.86.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:52.219 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-63 2021-03-12T02:49:52.219 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-63 zones={test-zone2, test-zone1} 2021-03-12T02:49:53.281 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:49:53.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:53.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:53.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:53.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:53.289 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:49:53.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:53.298 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:53.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:53.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:53.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:53.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:53.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:53.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:53.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:53.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:53.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:53.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:54.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:54.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:54.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:54.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:54.427 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-63 zones={test-zone2, test-zone1} 2021-03-12T02:49:54.427 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-63 zones={test-zone3, test-zone1} 2021-03-12T02:49:55.288 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:49:55.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:55.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:55.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:55.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:55.296 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:49:55.301 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:55.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:55.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:55.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:55.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:55.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:55.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:55.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:55.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:55.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:55.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:55.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:56.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:56.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:56.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:56.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:56.475 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-63 zones={test-zone3, test-zone1} 2021-03-12T02:49:56.475 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-63 zones={test-zone3, test-zone2} 2021-03-12T02:49:57.249 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:49:57.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:57.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:57.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:57.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:57.260 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:49:57.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:49:57.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:49:57.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:49:57.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:49:57.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:57.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:57.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:49:57.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:57.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:57.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:49:57.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:57.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:57.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:49:58.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:58.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:58.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:49:58.356 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-63 zones={test-zone3, test-zone2} 2021-03-12T02:49:58.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:58.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:58.359 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:58.403 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:58.404 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:58.632 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.83.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.63.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.38.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:58.633 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-64 2021-03-12T02:49:58.634 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:58.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:58.635 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:58.677 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:58.678 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:58.912 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:58.913 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-64 2021-03-12T02:49:58.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:58.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:58.914 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:58.955 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:58.956 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:59.195 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.83.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.63.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.38.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:59.196 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-64 2021-03-12T02:49:59.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:59.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:59.197 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:59.278 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:59.279 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:59.524 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:49:59.524 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-64 2021-03-12T02:49:59.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:59.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:59.526 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:59.609 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:59.610 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:49:59.832 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.83.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.63.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.38.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:49:59.833 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-64 2021-03-12T02:49:59.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:59.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:49:59.835 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:49:59.880 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:49:59.880 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:00.145 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.83.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.49.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.63.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.38.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:50:00.145 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-64 2021-03-12T02:50:00.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:00.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:00.147 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:00.188 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:00.189 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:00.417 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000001.77.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.53.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.58.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.29.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:50:00.418 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-64 2021-03-12T02:50:00.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:00.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:00.419 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:00.463 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:00.463 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:00.723 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.77.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.29.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:50:00.724 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-64 2021-03-12T02:50:00.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:00.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:00.725 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:00.767 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:00.768 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:00.997 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000001.77.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.53.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.58.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.29.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:50:00.998 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-64 2021-03-12T02:50:00.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:01.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:01.001 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:01.045 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:01.046 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:01.276 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.77.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.29.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:50:01.277 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-64 2021-03-12T02:50:01.278 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:01.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:01.280 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:01.367 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:01.368 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:01.599 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.90.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.44.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.58.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.36.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:50:01.600 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-64 2021-03-12T02:50:01.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:01.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:01.601 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:01.686 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:01.686 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:01.944 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.90.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.36.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:50:01.945 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-64 2021-03-12T02:50:01.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:01.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:01.946 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:01.988 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:01.988 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:02.227 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000001.90.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.44.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.58.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.36.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:50:02.228 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-64 2021-03-12T02:50:02.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:02.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:02.229 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:02.273 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:02.274 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:02.497 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.90.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.36.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:50:02.498 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-64 2021-03-12T02:50:02.498 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-64 zones={test-zone2, test-zone1} 2021-03-12T02:50:03.234 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:50:03.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:50:03.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:50:03.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:50:03.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:50:03.243 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:50:03.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:50:03.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:50:03.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:50:03.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:50:03.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:50:03.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:50:03.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:50:03.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:50:03.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:50:03.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:50:03.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:50:03.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:50:03.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:50:04.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:50:04.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:50:04.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:50:04.362 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-64 zones={test-zone2, test-zone1} 2021-03-12T02:50:04.363 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-64 zones={test-zone3, test-zone1} 2021-03-12T02:50:04.706 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:50:04.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:50:04.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:50:04.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:50:04.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:50:04.714 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:50:04.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:50:04.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:50:04.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:50:04.720 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:50:04.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:50:04.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:50:04.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:50:05.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:50:05.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:50:05.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:50:05.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:50:05.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:50:05.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:50:05.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:50:05.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:50:05.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:50:05.827 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-64 zones={test-zone3, test-zone1} 2021-03-12T02:50:05.827 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-64 zones={test-zone3, test-zone2} 2021-03-12T02:50:06.250 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:50:06.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:50:06.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:50:06.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:50:06.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:50:06.260 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:50:06.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:50:06.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:50:06.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:50:06.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:50:06.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:50:06.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:50:06.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:50:06.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:50:06.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:50:06.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:50:06.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:50:06.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:50:07.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:50:07.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:50:07.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:50:07.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:50:07.457 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-64 zones={test-zone3, test-zone2} 2021-03-12T02:50:07.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'bbucnh-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:50:07.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'bbucnh-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:50:07.459 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket bbucnh-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:50:07.503 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:07.504 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:07.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'bbucnh-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:50:07.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'bbucnh-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:50:07.792 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket bbucnh-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:50:07.873 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:07.874 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:08.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'bbucnh-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:50:08.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'bbucnh-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:50:08.149 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket bbucnh-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:50:08.231 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:08.232 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:08.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:08.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:08.513 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:08.592 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:08.593 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:08.789 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:50:08.822 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517408.469254_1177.1","last_update":"2021-03-12T02:50:08.469254Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:50:08.822 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:50:08.823 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:08.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:08.824 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:08.865 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:08.865 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:09.101 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517375.751736_1107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:49:35.751736Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:50:09.102 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615517408.469254_1177.1 target=1_1615517375.751736_1107.1 2021-03-12T02:50:14.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:14.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:14.104 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:14.146 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:14.147 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:14.359 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517408.469254_1177.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:08.469254Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:50:14.360 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:50:14.360 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:50:14.361 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:14.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:14.362 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:14.405 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:14.405 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:14.621 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517375.751736_1107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:49:35.751736Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:50:14.622 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1615517408.469254_1177.1 target=1_1615517375.751736_1107.1 2021-03-12T02:50:19.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:19.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:19.624 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:19.665 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:19.665 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:19.875 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517408.469254_1177.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:08.469254Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:50:19.875 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:50:19.876 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:50:19.876 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:19.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:19.877 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:19.917 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:19.918 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:20.177 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517408.469254_1177.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:08.469254Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:50:20.178 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:50:22.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'bbucnh-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:50:22.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'bbucnh-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:50:22.345 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket bbucnh-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:50:22.386 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:22.386 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:22.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'bbucnh-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:50:22.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'bbucnh-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:50:22.656 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket bbucnh-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:50:22.743 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:22.744 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:22.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'bbucnh-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:50:22.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'bbucnh-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:50:23.000 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket bbucnh-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:50:23.038 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:23.039 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:23.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:23.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:23.311 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:23.396 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:23.396 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:23.619 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.60.1"},{"key":1,"val":"00000000004.92.1"},{"key":2,"val":"00000000006.56.1"},{"key":3,"val":"00000000004.93.1"},{"key":4,"val":"00000000006.51.1"},{"key":5,"val":"00000000004.41.1"},{"key":6,"val":"00000000004.68.1"},{"key":7,"val":"00000000006.78.1"},{"key":8,"val":"00000000004.60.1"},{"key":9,"val":"00000000004.66.1"},{"key":10,"val":"00000000004.79.1"}]} 2021-03-12T02:50:23.619 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-62 2021-03-12T02:50:23.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:23.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:23.621 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:23.665 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:23.666 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:23.908 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.78.1"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:50:23.909 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone2 behind zone test-zone1: shard=0 master=00000000006.60.1 target=, shard=1 master=00000000004.92.1 target=, shard=2 master=00000000006.56.1 target=, shard=4 master=00000000006.51.1 target=, shard=5 master=00000000004.41.1 target=, shard=6 master=00000000004.68.1 target=, shard=8 master=00000000004.60.1 target=, shard=9 master=00000000004.66.1 target=, shard=10 master=00000000004.79.1 target= 2021-03-12T02:50:28.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:28.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:28.912 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:28.958 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:28.959 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:29.189 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.60.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.56.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.51.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.78.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:50:29.189 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-62 2021-03-12T02:50:29.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:29.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:29.191 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:29.234 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:29.235 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:29.451 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.60.1"},{"key":1,"val":"00000000004.92.1"},{"key":2,"val":"00000000006.56.1"},{"key":3,"val":"00000000004.93.1"},{"key":4,"val":"00000000006.51.1"},{"key":5,"val":"00000000004.41.1"},{"key":6,"val":"00000000004.68.1"},{"key":7,"val":"00000000006.78.1"},{"key":8,"val":"00000000004.60.1"},{"key":9,"val":"00000000004.66.1"},{"key":10,"val":"00000000004.79.1"}]} 2021-03-12T02:50:29.451 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-62 2021-03-12T02:50:29.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:29.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:29.453 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:29.536 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:29.537 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:29.776 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.78.1"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:50:29.777 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.60.1 target=, shard=1 master=00000000004.92.1 target=, shard=2 master=00000000006.56.1 target=, shard=4 master=00000000006.51.1 target=, shard=5 master=00000000004.41.1 target=, shard=6 master=00000000004.68.1 target=, shard=8 master=00000000004.60.1 target=, shard=9 master=00000000004.66.1 target=, shard=10 master=00000000004.79.1 target= 2021-03-12T02:50:34.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:34.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:34.779 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:34.821 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:34.821 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:35.019 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.60.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.56.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.51.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.78.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:50:35.020 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-62 2021-03-12T02:50:35.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:35.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:35.021 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:35.103 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:35.103 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:35.302 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.60.1"},{"key":1,"val":"00000000004.92.1"},{"key":2,"val":"00000000006.56.1"},{"key":3,"val":"00000000004.93.1"},{"key":4,"val":"00000000006.51.1"},{"key":5,"val":"00000000004.41.1"},{"key":6,"val":"00000000004.68.1"},{"key":7,"val":"00000000006.78.1"},{"key":8,"val":"00000000004.60.1"},{"key":9,"val":"00000000004.66.1"},{"key":10,"val":"00000000004.79.1"}]} 2021-03-12T02:50:35.302 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-62 2021-03-12T02:50:35.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:35.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:35.304 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:35.348 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:35.349 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:35.615 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:50:35.616 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:50:40.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:40.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:40.618 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:40.661 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:40.662 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:40.929 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:50:40.929 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:50:45.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:45.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:45.932 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:45.977 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:45.978 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:46.242 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:50:46.242 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:50:51.244 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:51.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:51.245 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:51.289 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:51.289 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:51.553 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:50:51.554 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:50:56.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:56.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:50:56.557 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:50:56.601 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:50:56.601 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:50:56.860 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:50:56.861 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:51:01.862 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:01.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:01.863 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:51:01.909 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:51:01.909 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:51:02.173 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:51:02.174 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:51:07.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:07.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:07.177 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:51:07.220 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:51:07.221 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:51:07.482 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:51:07.483 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:51:12.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:12.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:12.486 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:51:12.529 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:51:12.529 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:51:12.797 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:51:12.797 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:51:17.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:17.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:17.800 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:51:17.844 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:51:17.844 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:51:18.104 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:51:18.105 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:51:23.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:23.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:23.108 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:51:23.152 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:51:23.153 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:51:23.419 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:51:23.420 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:51:28.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:28.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:28.423 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:51:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:51:28.466 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:51:28.724 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:51:28.725 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:51:33.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:33.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:33.727 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:51:33.773 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:51:33.773 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:51:34.034 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:51:34.035 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:51:39.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:39.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:39.037 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:51:39.082 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:51:39.083 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:51:39.346 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:51:39.347 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:51:44.349 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:44.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:44.350 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:51:44.400 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:51:44.403 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:51:44.664 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:51:44.665 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:51:49.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:49.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:49.668 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:51:49.712 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:51:49.713 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:51:49.978 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:51:49.979 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:51:54.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:54.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:51:54.982 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:51:55.025 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:51:55.026 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:51:55.289 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:51:55.290 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:52:00.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:00.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:00.293 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:52:00.342 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:52:00.342 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:52:00.605 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:52:00.605 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:52:05.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:05.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:05.609 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:52:05.653 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:52:05.653 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:52:05.922 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:52:05.923 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:52:10.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:10.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:10.927 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:52:10.971 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:52:10.972 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:52:11.231 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:52:11.232 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:52:16.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:16.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:16.236 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:52:16.280 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:52:16.280 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:52:16.546 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:52:16.546 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:52:21.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:21.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:21.549 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:52:21.593 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:52:21.594 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:52:21.855 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:52:21.856 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:52:26.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:26.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:26.859 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:52:26.905 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:52:26.906 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:52:27.265 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:52:27.266 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1, shard=7 master=00000000006.78.1 target=00000000002.71.1 2021-03-12T02:52:32.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:32.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:32.268 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:52:32.312 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:52:32.313 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:52:32.573 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.78.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:52:32.573 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1 2021-03-12T02:52:37.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:37.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:37.576 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:52:37.621 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:52:37.622 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:52:37.881 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.78.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:52:37.882 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1 2021-03-12T02:52:42.883 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:42.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:42.885 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:52:42.928 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:52:42.929 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:52:43.187 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.78.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:52:43.187 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1 2021-03-12T02:52:48.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:48.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:48.190 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:52:48.238 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:52:48.238 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:52:48.505 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.78.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:52:48.506 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1 2021-03-12T02:52:53.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:53.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:53.509 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:52:53.553 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:52:53.554 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:52:53.818 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.78.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:52:53.819 WARNING:rgw_multi.tests:bucket bbucnh-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.60.1 target=00000000002.51.1, shard=2 master=00000000006.56.1 target=00000000002.49.1, shard=4 master=00000000006.51.1 target=00000000002.46.1 2021-03-12T02:52:58.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:58.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:58.822 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:52:58.867 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:52:58.868 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:52:59.136 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.60.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.56.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.51.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.78.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.60.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.66.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}}] 2021-03-12T02:52:59.136 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-62 2021-03-12T02:52:59.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:59.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:59.138 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:52:59.220 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:52:59.220 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:52:59.451 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000004.61.1"},{"key":1,"val":"00000000002.80.1"},{"key":2,"val":"00000000004.53.1"},{"key":3,"val":"00000000002.83.1"},{"key":4,"val":"00000000004.49.1"},{"key":5,"val":"00000000002.36.1"},{"key":6,"val":"00000000002.48.1"},{"key":7,"val":"00000000004.70.1"},{"key":8,"val":"00000000002.56.1"},{"key":9,"val":"00000000002.62.1"},{"key":10,"val":"00000000002.81.1"}]} 2021-03-12T02:52:59.452 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-62 2021-03-12T02:52:59.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:59.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:59.453 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:52:59.498 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:52:59.499 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:52:59.764 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.61.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.53.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.83.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.49.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.36.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.48.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.70.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.56.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.62.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.81.1"}}}] 2021-03-12T02:52:59.765 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-62 2021-03-12T02:52:59.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:59.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:52:59.767 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:52:59.807 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:52:59.807 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:00.021 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000004.61.1"},{"key":1,"val":"00000000002.80.1"},{"key":2,"val":"00000000004.53.1"},{"key":3,"val":"00000000002.83.1"},{"key":4,"val":"00000000004.49.1"},{"key":5,"val":"00000000002.36.1"},{"key":6,"val":"00000000002.48.1"},{"key":7,"val":"00000000004.70.1"},{"key":8,"val":"00000000002.56.1"},{"key":9,"val":"00000000002.62.1"},{"key":10,"val":"00000000002.81.1"}]} 2021-03-12T02:53:00.022 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-62 2021-03-12T02:53:00.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:00.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:00.024 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:00.067 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:00.068 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:00.296 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.61.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.53.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.83.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.49.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.36.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.48.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.70.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.56.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.62.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.81.1"}}}] 2021-03-12T02:53:00.296 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-62 2021-03-12T02:53:00.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:00.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:00.298 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:00.378 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:00.378 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:00.598 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000004.70.1"},{"key":1,"val":"00000000002.77.1"},{"key":2,"val":"00000000004.53.1"},{"key":3,"val":"00000000002.80.1"},{"key":4,"val":"00000000004.53.1"},{"key":5,"val":"00000000002.42.1"},{"key":6,"val":"00000000002.45.1"},{"key":7,"val":"00000000004.84.1"},{"key":8,"val":"00000000002.58.1"},{"key":9,"val":"00000000002.71.1"},{"key":10,"val":"00000000002.71.1"}]} 2021-03-12T02:53:00.598 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-62 2021-03-12T02:53:00.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:00.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:00.600 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:00.686 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:00.687 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:00.960 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.70.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.77.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.53.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.53.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.42.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.84.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.58.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}}] 2021-03-12T02:53:00.961 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-62 2021-03-12T02:53:00.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:00.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:00.962 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:01.002 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:01.002 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:01.227 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000004.70.1"},{"key":1,"val":"00000000002.77.1"},{"key":2,"val":"00000000004.53.1"},{"key":3,"val":"00000000002.80.1"},{"key":4,"val":"00000000004.53.1"},{"key":5,"val":"00000000002.42.1"},{"key":6,"val":"00000000002.45.1"},{"key":7,"val":"00000000004.84.1"},{"key":8,"val":"00000000002.58.1"},{"key":9,"val":"00000000002.71.1"},{"key":10,"val":"00000000002.71.1"}]} 2021-03-12T02:53:01.228 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-62 2021-03-12T02:53:01.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:01.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:01.229 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:01.273 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:01.273 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:01.514 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.70.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.77.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.53.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.53.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.42.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.84.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.58.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}}] 2021-03-12T02:53:01.515 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-62 2021-03-12T02:53:01.516 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-62 zones={test-zone2, test-zone1} 2021-03-12T02:53:01.530 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:53:01.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:01.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:01.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:01.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:01.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:01.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:01.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:01.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:01.544 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:53:01.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:01.553 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:01.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:01.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:01.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:01.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:01.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:01.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:01.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:01.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:01.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:01.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:01.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:02.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:02.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:02.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:02.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:02.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:02.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:02.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:02.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:02.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:02.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:02.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:03.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:03.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:03.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:03.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:03.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:03.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:03.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:03.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:03.791 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-62 zones={test-zone2, test-zone1} 2021-03-12T02:53:03.791 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-62 zones={test-zone3, test-zone1} 2021-03-12T02:53:03.800 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:53:03.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:03.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:03.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:03.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:03.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:03.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:03.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:03.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:03.809 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:53:03.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:03.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:03.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:03.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:03.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:03.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:03.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:03.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:03.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:03.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:04.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:04.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:04.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:04.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:04.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:04.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:04.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:04.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:04.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:04.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:04.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:04.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:05.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:05.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:05.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:05.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:05.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:05.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:05.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:05.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:05.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:05.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:06.015 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-62 zones={test-zone3, test-zone1} 2021-03-12T02:53:06.016 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-62 zones={test-zone3, test-zone2} 2021-03-12T02:53:06.069 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:53:06.077 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:06.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:06.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:06.078 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:06.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:06.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:06.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:06.080 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:06.080 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:53:06.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:06.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:06.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:06.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:06.088 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:06.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:06.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:06.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:06.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:06.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:06.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:06.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:06.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:06.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:06.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:06.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:06.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:06.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:06.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:07.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:07.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:07.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:07.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:07.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:07.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:07.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:07.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:07.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:07.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:08.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:08.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:08.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:08.332 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-62 zones={test-zone3, test-zone2} 2021-03-12T02:53:08.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:08.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:08.334 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:08.375 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:08.375 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:08.604 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.62.5"},{"key":1,"val":"00000000004.74.1"},{"key":2,"val":"00000000006.59.5"},{"key":3,"val":"00000000004.96.1"},{"key":4,"val":"00000000006.84.5"},{"key":5,"val":"00000000004.69.1"},{"key":6,"val":"00000000004.97.1"},{"key":7,"val":"00000000006.83.5"},{"key":8,"val":"00000000004.58.1"},{"key":9,"val":"00000000004.70.1"},{"key":10,"val":"00000000004.62.1"}]} 2021-03-12T02:53:08.604 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-63 2021-03-12T02:53:08.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:08.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:08.606 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:08.653 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:08.653 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:08.875 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.62.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.59.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.96.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.84.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.97.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.83.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}}] 2021-03-12T02:53:08.875 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-63 2021-03-12T02:53:08.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:08.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:08.877 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:08.920 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:08.920 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:09.146 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.62.5"},{"key":1,"val":"00000000004.74.1"},{"key":2,"val":"00000000006.59.5"},{"key":3,"val":"00000000004.96.1"},{"key":4,"val":"00000000006.84.5"},{"key":5,"val":"00000000004.69.1"},{"key":6,"val":"00000000004.97.1"},{"key":7,"val":"00000000006.83.5"},{"key":8,"val":"00000000004.58.1"},{"key":9,"val":"00000000004.70.1"},{"key":10,"val":"00000000004.62.1"}]} 2021-03-12T02:53:09.146 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-63 2021-03-12T02:53:09.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:09.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:09.151 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:09.233 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:09.233 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:09.476 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.62.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.59.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.96.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.84.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.97.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.83.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}}] 2021-03-12T02:53:09.477 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-63 2021-03-12T02:53:09.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:09.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:09.479 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:09.514 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:09.514 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:09.741 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.62.5"},{"key":1,"val":"00000000004.74.1"},{"key":2,"val":"00000000006.59.5"},{"key":3,"val":"00000000004.96.1"},{"key":4,"val":"00000000006.84.5"},{"key":5,"val":"00000000004.69.1"},{"key":6,"val":"00000000004.97.1"},{"key":7,"val":"00000000006.83.5"},{"key":8,"val":"00000000004.58.1"},{"key":9,"val":"00000000004.70.1"},{"key":10,"val":"00000000004.62.1"}]} 2021-03-12T02:53:09.742 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-63 2021-03-12T02:53:09.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:09.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:09.743 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:09.787 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:09.788 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:10.049 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.62.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.74.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.59.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.96.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.84.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.97.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.83.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.70.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}}] 2021-03-12T02:53:10.050 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-63 2021-03-12T02:53:10.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:10.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:10.052 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:10.132 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:10.132 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:10.343 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000004.59.1"},{"key":1,"val":"00000000002.54.1"},{"key":2,"val":"00000000004.60.1"},{"key":3,"val":"00000000002.74.1"},{"key":4,"val":"00000000004.100.1"},{"key":5,"val":"00000000002.61.1"},{"key":6,"val":"00000000002.79.1"},{"key":7,"val":"00000000004.80.1"},{"key":8,"val":"00000000002.52.1"},{"key":9,"val":"00000000002.45.1"},{"key":10,"val":"00000000002.60.1"}]} 2021-03-12T02:53:10.343 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-63 2021-03-12T02:53:10.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:10.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:10.345 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:10.391 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:10.391 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:10.635 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.59.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.60.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.74.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.100.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.79.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.80.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.52.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.60.1"}}}] 2021-03-12T02:53:10.636 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-63 2021-03-12T02:53:10.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:10.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:10.637 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:10.680 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:10.680 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:10.893 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000004.59.1"},{"key":1,"val":"00000000002.54.1"},{"key":2,"val":"00000000004.60.1"},{"key":3,"val":"00000000002.74.1"},{"key":4,"val":"00000000004.100.1"},{"key":5,"val":"00000000002.61.1"},{"key":6,"val":"00000000002.79.1"},{"key":7,"val":"00000000004.80.1"},{"key":8,"val":"00000000002.52.1"},{"key":9,"val":"00000000002.45.1"},{"key":10,"val":"00000000002.60.1"}]} 2021-03-12T02:53:10.894 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-63 2021-03-12T02:53:10.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:10.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:10.895 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:10.939 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:11.179 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.59.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.60.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.74.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.100.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.79.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.80.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.52.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.60.1"}}}] 2021-03-12T02:53:11.179 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-63 2021-03-12T02:53:11.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:11.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:11.181 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:11.261 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:11.262 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:11.480 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000005.64.1"},{"key":1,"val":"00000000002.68.1"},{"key":2,"val":"00000000004.57.1"},{"key":3,"val":"00000000002.73.1"},{"key":4,"val":"00000000004.91.1"},{"key":5,"val":"00000000002.67.1"},{"key":6,"val":"00000000002.76.1"},{"key":7,"val":"00000000004.70.1"},{"key":8,"val":"00000000002.52.1"},{"key":9,"val":"00000000002.43.1"},{"key":10,"val":"00000000002.69.1"}]} 2021-03-12T02:53:11.480 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-63 2021-03-12T02:53:11.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:11.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:11.482 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:11.563 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:11.563 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:11.811 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000005.64.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.57.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.73.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.91.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.76.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.70.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.52.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.43.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}}] 2021-03-12T02:53:11.812 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-63 2021-03-12T02:53:11.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:11.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:11.814 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:11.896 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:11.896 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:12.107 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000005.64.1"},{"key":1,"val":"00000000002.68.1"},{"key":2,"val":"00000000004.57.1"},{"key":3,"val":"00000000002.73.1"},{"key":4,"val":"00000000004.91.1"},{"key":5,"val":"00000000002.67.1"},{"key":6,"val":"00000000002.76.1"},{"key":7,"val":"00000000004.70.1"},{"key":8,"val":"00000000002.52.1"},{"key":9,"val":"00000000002.43.1"},{"key":10,"val":"00000000002.69.1"}]} 2021-03-12T02:53:12.107 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-63 2021-03-12T02:53:12.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:12.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:12.109 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:12.154 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:12.154 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:12.374 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000005.64.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.57.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.73.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.91.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.76.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.70.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.52.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.43.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}}] 2021-03-12T02:53:12.374 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-63 2021-03-12T02:53:12.375 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-63 zones={test-zone2, test-zone1} 2021-03-12T02:53:13.882 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:53:13.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:13.889 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:13.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:13.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:13.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:13.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:13.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:13.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:13.892 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:53:13.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:13.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:13.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:13.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:13.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:13.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:13.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:13.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:13.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:13.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:14.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:14.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:14.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:14.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:14.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:14.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:14.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:14.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:14.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:14.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:15.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:15.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:15.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:15.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:15.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:15.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:15.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:15.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:15.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:15.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:15.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:16.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:16.140 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-63 zones={test-zone2, test-zone1} 2021-03-12T02:53:16.141 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-63 zones={test-zone3, test-zone1} 2021-03-12T02:53:16.580 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:53:16.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:16.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:16.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:16.587 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:16.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:16.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:16.588 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:16.589 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:16.589 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:53:16.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:16.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:16.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:16.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:16.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:16.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:16.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:16.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:16.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:16.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:16.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:16.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:16.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:17.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:17.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:17.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:17.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:17.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:17.538 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:17.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:17.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:17.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:17.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:18.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:18.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:18.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:18.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:18.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:18.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:18.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:18.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:18.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:18.881 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-63 zones={test-zone3, test-zone1} 2021-03-12T02:53:18.882 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-63 zones={test-zone3, test-zone2} 2021-03-12T02:53:19.763 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:53:19.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:19.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:19.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:19.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:19.774 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:19.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:19.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:19.776 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:19.777 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:53:19.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:19.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:19.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:19.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:19.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:19.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:19.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:19.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:19.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:19.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:19.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:20.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:20.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:20.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:20.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:20.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:20.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:20.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:20.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:20.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:20.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:20.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:21.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:21.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:21.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:21.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:21.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:21.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:21.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:21.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:21.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:21.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:21.980 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-63 zones={test-zone3, test-zone2} 2021-03-12T02:53:21.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:21.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:21.983 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:22.028 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:22.028 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:22.248 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.89.5"},{"key":1,"val":"00000000004.67.1"},{"key":2,"val":"00000000006.61.5"},{"key":3,"val":"00000000004.99.1"},{"key":4,"val":"00000000006.73.5"},{"key":5,"val":"00000000004.76.1"},{"key":6,"val":"00000000004.64.1"},{"key":7,"val":"00000000006.44.5"},{"key":8,"val":"00000000004.79.1"},{"key":9,"val":"00000000004.80.1"},{"key":10,"val":"00000000004.68.1"}]} 2021-03-12T02:53:22.249 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-64 2021-03-12T02:53:22.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:22.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:22.250 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:22.295 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:22.296 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:22.521 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.89.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.61.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.99.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.73.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.44.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}}] 2021-03-12T02:53:22.522 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-64 2021-03-12T02:53:22.523 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:22.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:22.524 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:22.588 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:22.591 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:22.806 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.89.5"},{"key":1,"val":"00000000004.67.1"},{"key":2,"val":"00000000006.61.5"},{"key":3,"val":"00000000004.99.1"},{"key":4,"val":"00000000006.73.5"},{"key":5,"val":"00000000004.76.1"},{"key":6,"val":"00000000004.64.1"},{"key":7,"val":"00000000006.44.5"},{"key":8,"val":"00000000004.79.1"},{"key":9,"val":"00000000004.80.1"},{"key":10,"val":"00000000004.68.1"}]} 2021-03-12T02:53:22.806 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-64 2021-03-12T02:53:22.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:22.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:22.808 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:22.909 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:22.909 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:23.153 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.89.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.61.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.99.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.73.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.44.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}}] 2021-03-12T02:53:23.154 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-64 2021-03-12T02:53:23.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:23.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:23.155 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:23.239 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:23.240 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:23.452 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000006.89.5"},{"key":1,"val":"00000000004.67.1"},{"key":2,"val":"00000000006.61.5"},{"key":3,"val":"00000000004.99.1"},{"key":4,"val":"00000000006.73.5"},{"key":5,"val":"00000000004.76.1"},{"key":6,"val":"00000000004.64.1"},{"key":7,"val":"00000000006.44.5"},{"key":8,"val":"00000000004.79.1"},{"key":9,"val":"00000000004.80.1"},{"key":10,"val":"00000000004.68.1"}]} 2021-03-12T02:53:23.453 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-64 2021-03-12T02:53:23.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:23.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:23.454 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:23.498 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:23.498 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:23.756 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.89.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.61.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.99.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.73.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.44.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}}] 2021-03-12T02:53:23.757 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-64 2021-03-12T02:53:23.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:23.758 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:23.759 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:23.842 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:23.843 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:24.058 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000004.83.5"},{"key":1,"val":"00000000002.63.1"},{"key":2,"val":"00000000004.65.5"},{"key":3,"val":"00000000002.113.1"},{"key":4,"val":"00000000004.68.5"},{"key":5,"val":"00000000002.57.1"},{"key":6,"val":"00000000002.59.1"},{"key":7,"val":"00000000004.35.5"},{"key":8,"val":"00000000002.71.1"},{"key":9,"val":"00000000002.69.1"},{"key":10,"val":"00000000002.59.1"}]} 2021-03-12T02:53:24.059 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-64 2021-03-12T02:53:24.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:24.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:24.060 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:24.106 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:24.107 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:24.345 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.83.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.65.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.113.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.68.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.57.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.59.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.35.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.59.1"}}}] 2021-03-12T02:53:24.345 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-64 2021-03-12T02:53:24.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:24.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:24.347 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:24.389 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:24.389 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:24.623 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":"00000000004.83.5"},{"key":1,"val":"00000000002.63.1"},{"key":2,"val":"00000000004.65.5"},{"key":3,"val":"00000000002.113.1"},{"key":4,"val":"00000000004.68.5"},{"key":5,"val":"00000000002.57.1"},{"key":6,"val":"00000000002.59.1"},{"key":7,"val":"00000000004.35.5"},{"key":8,"val":"00000000002.71.1"},{"key":9,"val":"00000000002.69.1"},{"key":10,"val":"00000000002.59.1"}]} 2021-03-12T02:53:24.624 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-64 2021-03-12T02:53:24.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:24.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:24.626 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:24.671 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:24.672 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:24.911 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.83.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.65.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.113.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.68.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.57.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.59.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.35.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.59.1"}}}] 2021-03-12T02:53:24.912 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-64 2021-03-12T02:53:24.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:24.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:24.914 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:24.996 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:24.996 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:25.227 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000004.95.1"},{"key":1,"val":"00000000002.82.1"},{"key":2,"val":"00000000004.56.1"},{"key":3,"val":"00000000002.108.1"},{"key":4,"val":"00000000004.66.1"},{"key":5,"val":"00000000002.65.1"},{"key":6,"val":"00000000002.68.1"},{"key":7,"val":"00000000004.41.1"},{"key":8,"val":"00000000002.82.1"},{"key":9,"val":"00000000002.83.1"},{"key":10,"val":"00000000002.69.1"}]} 2021-03-12T02:53:25.227 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-64 2021-03-12T02:53:25.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:25.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:25.229 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:25.310 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:25.310 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:25.553 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.95.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.82.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.56.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.108.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.66.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.41.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.82.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.83.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}}] 2021-03-12T02:53:25.553 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-64 2021-03-12T02:53:25.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:25.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:25.555 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:25.640 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:25.640 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:25.869 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":"00000000004.95.1"},{"key":1,"val":"00000000002.82.1"},{"key":2,"val":"00000000004.56.1"},{"key":3,"val":"00000000002.108.1"},{"key":4,"val":"00000000004.66.1"},{"key":5,"val":"00000000002.65.1"},{"key":6,"val":"00000000002.68.1"},{"key":7,"val":"00000000004.41.1"},{"key":8,"val":"00000000002.82.1"},{"key":9,"val":"00000000002.83.1"},{"key":10,"val":"00000000002.69.1"}]} 2021-03-12T02:53:25.869 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-64 2021-03-12T02:53:25.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:25.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:25.871 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:25.916 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:25.916 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:26.142 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.95.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.82.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.56.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.108.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.66.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.65.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.41.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.82.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.83.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}}] 2021-03-12T02:53:26.142 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-64 2021-03-12T02:53:26.143 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-64 zones={test-zone2, test-zone1} 2021-03-12T02:53:27.618 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:53:27.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:27.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:27.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:27.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:27.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:27.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:27.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:27.628 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:27.628 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:53:27.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:27.636 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:27.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:27.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:27.638 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:27.638 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:27.638 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:27.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:27.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:27.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:27.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:27.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:27.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:28.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:28.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:28.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:28.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:28.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:28.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:28.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:28.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:28.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:28.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:29.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:29.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:29.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:29.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:29.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:29.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:29.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:29.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:29.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:29.914 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-64 zones={test-zone2, test-zone1} 2021-03-12T02:53:29.915 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-64 zones={test-zone3, test-zone1} 2021-03-12T02:53:30.230 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:53:30.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:30.237 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:30.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:30.238 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:30.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:30.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:30.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:30.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:30.240 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:53:30.247 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:30.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:30.249 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:30.251 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:30.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:30.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:30.253 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:30.254 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:30.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:30.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:30.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:30.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:30.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:30.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:30.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:30.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:30.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:31.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:31.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:31.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:31.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:31.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:31.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:31.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:31.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:31.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:31.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:32.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:32.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:32.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:32.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:32.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:32.510 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-64 zones={test-zone3, test-zone1} 2021-03-12T02:53:32.510 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-64 zones={test-zone3, test-zone2} 2021-03-12T02:53:33.442 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:53:33.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:33.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:33.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:33.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:33.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:33.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:33.454 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:33.455 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:33.455 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:53:33.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-12T02:53:33.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-12T02:53:33.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-12T02:53:33.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-12T02:53:33.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-12T02:53:33.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-12T02:53:33.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-12T02:53:33.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-12T02:53:33.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:33.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:33.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-12T02:53:33.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:33.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:33.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-12T02:53:34.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:34.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:34.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-12T02:53:34.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:34.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:34.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-12T02:53:34.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:34.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:34.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-12T02:53:34.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:34.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:35.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-12T02:53:35.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:35.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:35.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-12T02:53:35.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:35.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:35.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-12T02:53:35.692 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-64 zones={test-zone3, test-zone2} 2021-03-12T02:53:35.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:35.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:35.694 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:35.734 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:35.734 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:35.959 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615517436.038995_620.1","last_update":"2021-03-12T02:50:36.038995Z"},{"marker":"1_1615517436.080288_767.1","last_update":"2021-03-12T02:50:36.080288Z"},{"marker":"1_1615517423.275191_835.3","last_update":"2021-03-12T02:50:23.275191Z"},{"marker":"1_1615517436.047790_623.1","last_update":"2021-03-12T02:50:36.047790Z"}] 2021-03-12T02:53:35.960 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-12T02:53:35.960 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:35.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:35.961 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:36.006 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:36.006 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:36.223 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":8376578737807985417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517436.038995_620.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:36.038995Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517436.080288_767.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:36.080288Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517423.275191_835.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:23.275191Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517436.047790_623.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:36.047790Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:53:36.223 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-12T02:53:36.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:36.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:36.225 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:36.265 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:36.265 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:36.501 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615517436.038995_620.1","last_update":"2021-03-12T02:50:36.038995Z"},{"marker":"1_1615517436.080288_767.1","last_update":"2021-03-12T02:50:36.080288Z"},{"marker":"1_1615517423.275191_835.3","last_update":"2021-03-12T02:50:23.275191Z"},{"marker":"1_1615517436.047790_623.1","last_update":"2021-03-12T02:50:36.047790Z"}] 2021-03-12T02:53:36.502 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-12T02:53:36.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:36.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:36.503 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:36.584 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:36.584 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:36.807 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1279455150295362088},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517436.038995_620.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:36.038995Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517436.080288_767.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:36.080288Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517423.275191_835.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:23.275191Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517436.047790_623.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:36.047790Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:53:36.807 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-12T02:53:36.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:36.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:36.809 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:36.889 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:36.889 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:37.118 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615517436.038995_620.1","last_update":"2021-03-12T02:50:36.038995Z"},{"marker":"1_1615517436.080288_767.1","last_update":"2021-03-12T02:50:36.080288Z"},{"marker":"1_1615517423.275191_835.3","last_update":"2021-03-12T02:50:23.275191Z"},{"marker":"1_1615517436.047790_623.1","last_update":"2021-03-12T02:50:36.047790Z"}] 2021-03-12T02:53:37.119 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-12T02:53:37.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:37.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:37.121 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:37.163 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:37.163 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:37.427 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11911835907256974194},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517436.038995_620.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:36.038995Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517436.080288_767.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:36.080288Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517423.275191_835.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:23.275191Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517436.047790_623.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:36.047790Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:53:37.428 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-12T02:53:37.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:37.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:37.429 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:37.507 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:37.507 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:37.709 INFO:teuthology.orchestra.run.smithi114.stdout:[{"marker":"1_1615517449.264895_652.6","last_update":"2021-03-12T02:50:49.264895Z"},{"marker":"1_1615517449.264895_653.9","last_update":"2021-03-12T02:50:49.264895Z"},{"marker":"1_1615517449.264895_673.9","last_update":"2021-03-12T02:50:49.264895Z"},{"marker":"1_1615517449.264895_557.9","last_update":"2021-03-12T02:50:49.264895Z"}] 2021-03-12T02:53:37.709 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-12T02:53:37.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:37.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:37.711 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:37.755 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:37.755 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:38.000 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9936002905845503227},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517449.264895_652.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:49.264895Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517449.264895_653.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:49.264895Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517449.264895_673.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:49.264895Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517449.264895_557.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:49.264895Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:53:38.001 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-12T02:53:38.001 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:38.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:38.002 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:38.042 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:38.254 INFO:teuthology.orchestra.run.smithi114.stdout:[{"marker":"1_1615517449.264895_652.6","last_update":"2021-03-12T02:50:49.264895Z"},{"marker":"1_1615517449.264895_653.9","last_update":"2021-03-12T02:50:49.264895Z"},{"marker":"1_1615517449.264895_673.9","last_update":"2021-03-12T02:50:49.264895Z"},{"marker":"1_1615517449.264895_557.9","last_update":"2021-03-12T02:50:49.264895Z"}] 2021-03-12T02:53:38.254 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-12T02:53:38.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:38.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:38.256 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:38.298 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:38.299 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:38.526 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3923957601198887203},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517449.264895_652.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:49.264895Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517449.264895_653.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:49.264895Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517449.264895_673.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:49.264895Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517449.264895_557.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:49.264895Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:53:38.526 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-12T02:53:38.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:38.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:38.528 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:38.607 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:38.608 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:38.840 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615517443.593670_550.6","last_update":"2021-03-12T02:50:43.593670Z"},{"marker":"1_1615517443.593670_558.9","last_update":"2021-03-12T02:50:43.593670Z"},{"marker":"1_1615517443.593670_600.9","last_update":"2021-03-12T02:50:43.593670Z"},{"marker":"1_1615517443.593670_462.9","last_update":"2021-03-12T02:50:43.593670Z"}] 2021-03-12T02:53:38.841 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-12T02:53:38.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:38.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:38.842 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:38.926 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:38.926 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:39.160 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":12616054323329998860},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517443.593670_550.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:43.593670Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517443.593670_558.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:43.593670Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517443.593670_600.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:43.593670Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517443.593670_462.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:43.593670Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:53:39.160 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-12T02:53:39.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:39.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:39.162 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:39.238 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:39.239 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:39.469 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"1_1615517443.593670_550.6","last_update":"2021-03-12T02:50:43.593670Z"},{"marker":"1_1615517443.593670_558.9","last_update":"2021-03-12T02:50:43.593670Z"},{"marker":"1_1615517443.593670_600.9","last_update":"2021-03-12T02:50:43.593670Z"},{"marker":"1_1615517443.593670_462.9","last_update":"2021-03-12T02:50:43.593670Z"}] 2021-03-12T02:53:39.469 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-12T02:53:39.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:39.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:39.471 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:39.512 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:39.512 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:39.721 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":10754379456686351807},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1615517443.593670_550.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:43.593670Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1615517443.593670_558.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:43.593670Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1615517443.593670_600.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:43.593670Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1615517443.593670_462.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:50:43.593670Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-12T02:53:39.721 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-12T02:53:39.722 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2021-03-12T02:53:39.723 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-65 2021-03-12T02:53:40.287 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=bbucnh-66 2021-03-12T02:53:40.924 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=bbucnh-67 2021-03-12T02:53:43.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:43.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:43.092 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:43.137 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:43.137 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:43.396 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.99.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:53:43.397 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-65 2021-03-12T02:53:43.397 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:43.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:43.398 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:43.441 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:43.442 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:43.647 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:53:43.648 WARNING:rgw_multi.tests:bucket bbucnh-65 zone test-zone2 behind zone test-zone1: shard=5 master=00000000011.99.5 target= 2021-03-12T02:53:48.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:48.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:48.651 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:48.697 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:48.698 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:48.925 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000011.99.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:53:48.925 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-65 2021-03-12T02:53:48.926 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:48.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:48.927 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:48.967 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:48.968 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:49.195 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.99.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:53:49.195 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-65 2021-03-12T02:53:49.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:49.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:49.197 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:49.277 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:49.278 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:49.510 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000011.99.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:53:49.511 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-65 2021-03-12T02:53:49.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:49.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:49.513 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:49.592 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:49.592 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:49.825 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.99.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:53:49.825 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-65 2021-03-12T02:53:49.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:49.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:49.827 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:49.874 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:50.143 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.99.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:53:50.143 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-65 2021-03-12T02:53:50.144 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:50.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:50.145 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:50.186 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:50.186 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:50.397 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.104.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:53:50.398 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-65 2021-03-12T02:53:50.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:50.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:50.399 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:50.442 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:50.442 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:50.688 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.104.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:53:50.689 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-65 2021-03-12T02:53:50.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:50.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:50.690 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:50.733 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:50.733 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:50.947 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.104.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:53:50.947 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-65 2021-03-12T02:53:50.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:50.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:50.949 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:50.993 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:50.993 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:51.248 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.104.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:53:51.249 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-65 2021-03-12T02:53:51.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:51.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:51.250 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:51.337 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:51.338 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:51.582 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.95.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:53:51.582 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-65 2021-03-12T02:53:51.583 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:51.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:51.584 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:51.627 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:51.627 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:51.877 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.95.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:53:51.878 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-65 2021-03-12T02:53:51.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:51.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:51.880 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:51.961 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:51.961 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:52.190 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.95.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:53:52.190 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-65 2021-03-12T02:53:52.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:52.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:52.192 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:52.238 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:52.238 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:52.473 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.95.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:53:52.474 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-65 2021-03-12T02:53:52.474 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-65 zones={test-zone2, test-zone1} 2021-03-12T02:53:53.357 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:53:53.364 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-03-12T02:53:53.366 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:53:53.415 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-03-12T02:53:53.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-12T02:53:53.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-12T02:53:54.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-12T02:53:54.335 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-65 zones={test-zone2, test-zone1} 2021-03-12T02:53:54.335 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-65 zones={test-zone3, test-zone1} 2021-03-12T02:53:54.543 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:53:54.591 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-03-12T02:53:54.591 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:53:54.599 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-03-12T02:53:54.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-12T02:53:54.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-12T02:53:55.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-12T02:53:55.642 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-65 zones={test-zone3, test-zone1} 2021-03-12T02:53:55.643 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-65 zones={test-zone3, test-zone2} 2021-03-12T02:53:56.147 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:53:56.156 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-03-12T02:53:56.157 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:53:56.203 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-03-12T02:53:56.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-12T02:53:56.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-12T02:53:56.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-12T02:53:57.173 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-65 zones={test-zone3, test-zone2} 2021-03-12T02:53:57.174 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... ok 2021-03-12T02:53:57.175 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-68 2021-03-12T02:53:57.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:57.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:57.613 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:57.653 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:57.653 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:57.863 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:53:57.895 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517637.505778_1312.1","last_update":"2021-03-12T02:53:57.505778Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:53:57.895 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:53:57.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:57.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:57.897 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:57.938 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:57.938 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:58.152 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517637.505778_1312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:53:57.505778Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:53:58.153 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:53:58.154 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:53:58.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:58.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:58.155 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:58.195 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:58.195 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:58.431 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517637.505778_1312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:53:57.505778Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:53:58.432 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:53:58.433 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:53:58.434 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:58.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:58.436 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:58.476 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:58.477 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:58.834 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517637.505778_1312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:53:57.505778Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:53:58.837 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:53:58.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:58.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:58.839 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-68 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:58.888 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:58.889 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:59.120 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.80.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.43.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-12T02:53:59.121 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-68 2021-03-12T02:53:59.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:59.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:53:59.122 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-68 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:53:59.169 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:53:59.169 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:53:59.405 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"testobj-sse-c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.80.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"testobj-sse-kms","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:53:59.405 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-68 2021-03-12T02:54:00.366 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... ok 2021-03-12T02:54:00.367 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-69 2021-03-12T02:54:01.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:01.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:01.280 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:01.315 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:01.316 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:01.516 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:54:01.546 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517641.074173_1322.1","last_update":"2021-03-12T02:54:01.074173Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:54:01.547 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:54:01.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:01.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:01.548 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:01.587 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:01.587 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:01.794 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517641.074173_1322.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:01.074173Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:01.795 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:54:01.795 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:54:01.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:01.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:01.797 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:01.838 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:01.839 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:02.044 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517641.074173_1322.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:01.074173Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:02.045 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:54:02.045 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:54:02.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:02.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:02.047 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:02.091 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:02.359 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517641.074173_1322.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:01.074173Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:02.359 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:54:02.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:02.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:02.361 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:02.401 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:02.402 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:02.619 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.62.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.83.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.62.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.44.5"}]} 2021-03-12T02:54:02.620 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-69 2021-03-12T02:54:02.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:02.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:02.621 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:02.666 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:02.666 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:02.892 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}}] 2021-03-12T02:54:02.892 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-69 2021-03-12T02:54:02.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:02.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:02.894 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:02.936 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:02.936 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:03.149 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.62.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.83.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.62.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.44.5"}]} 2021-03-12T02:54:03.150 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-69 2021-03-12T02:54:03.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:03.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:03.151 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:03.236 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:03.237 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:03.475 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}}] 2021-03-12T02:54:03.476 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-69 2021-03-12T02:54:03.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:03.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:03.477 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:03.557 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:03.558 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:03.779 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.62.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.83.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.62.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.44.5"}]} 2021-03-12T02:54:03.779 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-69 2021-03-12T02:54:03.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:03.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:03.781 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:03.825 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:03.825 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:04.091 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.62.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.83.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.62.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.44.5"}}}] 2021-03-12T02:54:04.091 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-69 2021-03-12T02:54:04.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:04.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:04.093 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:04.177 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:04.177 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:04.384 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.56.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.75.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.60.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.41.5"}]} 2021-03-12T02:54:04.385 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-69 2021-03-12T02:54:04.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:04.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:04.387 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:04.669 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5"}}}] 2021-03-12T02:54:04.670 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-69 2021-03-12T02:54:04.670 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:04.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:04.671 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:04.713 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:04.714 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:04.922 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.56.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.75.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.60.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.41.5"}]} 2021-03-12T02:54:04.922 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-69 2021-03-12T02:54:04.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:04.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:04.924 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:04.970 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:04.970 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:05.213 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5"}}}] 2021-03-12T02:54:05.213 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-69 2021-03-12T02:54:05.214 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:05.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:05.215 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:05.299 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:05.299 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:05.514 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.68.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.89.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.59.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.47.5"}]} 2021-03-12T02:54:05.515 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-69 2021-03-12T02:54:05.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:05.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:05.517 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:05.600 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:05.601 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:05.847 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.89.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}}] 2021-03-12T02:54:05.848 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-69 2021-03-12T02:54:05.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:05.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:05.850 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:05.929 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:05.930 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:06.150 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.68.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.89.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.59.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.47.5"}]} 2021-03-12T02:54:06.151 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-69 2021-03-12T02:54:06.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:06.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:06.152 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:06.197 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:06.197 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:06.426 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.89.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}}] 2021-03-12T02:54:06.426 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-69 2021-03-12T02:54:06.427 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-69 zones={test-zone2, test-zone1} 2021-03-12T02:54:07.392 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:54:07.399 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-12T02:54:07.400 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-12T02:54:07.400 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-12T02:54:07.401 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-12T02:54:07.401 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:54:07.409 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-12T02:54:07.410 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-12T02:54:07.410 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-12T02:54:07.411 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-12T02:54:07.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:07.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:07.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:07.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:07.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:07.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:08.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:08.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:08.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:08.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:08.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:08.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:08.547 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-69 zones={test-zone2, test-zone1} 2021-03-12T02:54:08.547 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-69 zones={test-zone3, test-zone1} 2021-03-12T02:54:08.764 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:54:08.770 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-12T02:54:08.770 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-12T02:54:08.771 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-12T02:54:08.771 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-12T02:54:08.772 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:54:08.783 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-12T02:54:08.784 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-12T02:54:08.784 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-12T02:54:08.785 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-12T02:54:08.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:08.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:09.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:09.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:09.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:09.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:09.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:09.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:09.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:09.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:09.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:09.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:09.961 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-69 zones={test-zone3, test-zone1} 2021-03-12T02:54:09.962 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-69 zones={test-zone3, test-zone2} 2021-03-12T02:54:10.606 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:54:10.615 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-12T02:54:10.616 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-12T02:54:10.616 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-12T02:54:10.617 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-12T02:54:10.617 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:54:10.624 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-12T02:54:10.624 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-12T02:54:10.625 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-12T02:54:10.625 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-12T02:54:10.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:10.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:10.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:10.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:11.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:11.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:11.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:11.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:11.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:11.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:11.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:11.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:11.817 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-69 zones={test-zone3, test-zone2} 2021-03-12T02:54:11.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:54:11.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:54:11.818 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:54:11.855 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:11.856 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:12.096 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=bbucnh-70 2021-03-12T02:54:12.800 INFO:tasks.rgw.c1.client.1.smithi077.stdout:2021-03-12T02:54:12.793+0000 7eff1f7ee700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-12T02:54:12.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:12.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:12.931 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:12.973 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:12.974 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:13.162 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:54:13.193 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517652.719188_1347.1","last_update":"2021-03-12T02:54:12.719188Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:54:13.194 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:54:13.195 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:13.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:13.196 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:13.236 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:13.237 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:13.455 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517652.719188_1347.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:12.719188Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:13.456 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:54:13.457 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:54:13.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:13.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:13.458 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:13.498 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:13.498 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:13.728 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517641.074173_1322.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:01.074173Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:13.729 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1615517652.719188_1347.1 target=1_1615517641.074173_1322.1 2021-03-12T02:54:18.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:18.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:18.732 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:18.773 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:18.773 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:19.014 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517652.719188_1347.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:12.719188Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:19.014 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:54:19.015 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:54:19.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:19.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:19.017 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:19.059 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:19.059 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:19.325 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517652.719188_1347.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:12.719188Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:19.325 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:54:19.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:19.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:19.327 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:19.369 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:19.370 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:19.604 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.104.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.65.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.95.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.63.5"}]} 2021-03-12T02:54:19.605 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-70 2021-03-12T02:54:19.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:19.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:19.607 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:19.653 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:19.653 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:19.874 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.104.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.95.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}}] 2021-03-12T02:54:19.875 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=bbucnh-70 2021-03-12T02:54:19.875 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:19.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:19.876 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:19.918 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:19.918 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:20.158 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.104.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.65.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.95.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.63.5"}]} 2021-03-12T02:54:20.159 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-70 2021-03-12T02:54:20.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:20.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:20.161 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:20.246 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:20.246 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:20.485 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.104.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.95.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}}] 2021-03-12T02:54:20.486 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=bbucnh-70 2021-03-12T02:54:20.486 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:20.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:20.487 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:20.529 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:20.530 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:20.778 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.104.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.65.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.95.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.63.5"}]} 2021-03-12T02:54:20.779 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-70 2021-03-12T02:54:20.779 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:20.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:20.780 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:20.823 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:20.823 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:21.087 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.104.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.65.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.95.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.63.5"}}}] 2021-03-12T02:54:21.088 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-70 2021-03-12T02:54:21.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:21.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:21.090 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:21.173 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:21.173 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:21.397 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.122.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.59.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.89.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.77.5"}]} 2021-03-12T02:54:21.398 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-70 2021-03-12T02:54:21.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:21.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:21.399 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:21.443 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:21.443 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:21.692 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.122.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.89.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.77.5"}}}] 2021-03-12T02:54:21.693 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=bbucnh-70 2021-03-12T02:54:21.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:21.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:21.695 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:21.736 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:21.736 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:21.953 INFO:teuthology.orchestra.run.smithi114.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.122.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.59.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.89.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.77.5"}]} 2021-03-12T02:54:21.954 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-70 2021-03-12T02:54:21.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:21.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:21.955 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket bbucnh-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:22.000 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:22.000 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:22.243 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.122.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.89.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.77.5"}}}] 2021-03-12T02:54:22.244 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=bbucnh-70 2021-03-12T02:54:22.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:22.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:22.245 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:22.328 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:22.328 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:22.555 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.115.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.72.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.103.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.63.5"}]} 2021-03-12T02:54:22.556 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-70 2021-03-12T02:54:22.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:22.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:22.557 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:22.639 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:22.640 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:22.886 INFO:teuthology.orchestra.run.smithi077.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.115.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.103.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}}] 2021-03-12T02:54:22.887 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=bbucnh-70 2021-03-12T02:54:22.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:22.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:22.888 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:22.970 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:22.970 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:23.201 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.115.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.72.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.103.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.63.5"}]} 2021-03-12T02:54:23.202 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-70 2021-03-12T02:54:23.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:23.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'bbucnh-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:23.204 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket bbucnh-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:23.249 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:23.249 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:23.488 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.115.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.103.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}}] 2021-03-12T02:54:23.489 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=bbucnh-70 2021-03-12T02:54:23.489 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-70 zones={test-zone2, test-zone1} 2021-03-12T02:54:24.940 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:54:24.947 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-12T02:54:24.947 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-12T02:54:24.948 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-12T02:54:24.948 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-12T02:54:24.949 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:54:24.956 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-12T02:54:24.956 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-12T02:54:24.957 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-12T02:54:24.957 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-12T02:54:24.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:25.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:25.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:25.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:25.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:25.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:25.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:25.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:25.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:25.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:25.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:25.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:26.067 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-70 zones={test-zone2, test-zone1} 2021-03-12T02:54:26.067 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-70 zones={test-zone3, test-zone1} 2021-03-12T02:54:26.590 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:54:26.595 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-12T02:54:26.596 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-12T02:54:26.596 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-12T02:54:26.597 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-12T02:54:26.597 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:54:26.602 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-12T02:54:26.603 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-12T02:54:26.603 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-12T02:54:26.604 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-12T02:54:26.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:26.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:26.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:26.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:26.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:27.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:27.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:27.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:27.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:27.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:27.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:27.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:27.711 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-70 zones={test-zone3, test-zone1} 2021-03-12T02:54:27.711 INFO:tasks.rgw_multi.zone_rados:comparing bucket=bbucnh-70 zones={test-zone3, test-zone2} 2021-03-12T02:54:28.196 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-12T02:54:28.203 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-12T02:54:28.204 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-12T02:54:28.204 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-12T02:54:28.205 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-12T02:54:28.205 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-12T02:54:28.212 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-12T02:54:28.213 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-12T02:54:28.213 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-12T02:54:28.214 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-12T02:54:28.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:28.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:28.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-12T02:54:28.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:28.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:28.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-12T02:54:28.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:28.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:28.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-12T02:54:29.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:29.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:29.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-12T02:54:29.336 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=bbucnh-70 zones={test-zone3, test-zone2} 2021-03-12T02:54:29.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:54:29.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:54:29.338 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:54:29.375 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:29.375 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:29.634 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:29.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'bbucnh-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:29.635 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket bbucnh-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:29.713 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:29.713 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:29.946 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2021-03-12T02:54:29.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:29.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:29.948 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket bbucnh-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:30.027 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:30.028 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:30.262 INFO:teuthology.orchestra.run.smithi077.stdout:[{"op_id":"1#00000000001.62.5","op_tag":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6165.4503","op":"write","object":"a","instance":"","state":"complete","index_ver":1,"timestamp":"2021-03-12T02:54:01.084334169Z","ver":{"pool":9,"epoch":4},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-69:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.20"}]},{"op_id":"5#00000000001.83.5","op_tag":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6165.4509","op":"write","object":"d","instance":"","state":"complete","index_ver":1,"timestamp":"2021-03-12T02:54:01.231180144Z","ver":{"pool":9,"epoch":4},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-69:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.20"}]},{"op_id":"8#00000000001.62.5","op_tag":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6165.4507","op":"write","object":"c","instance":"","state":"complete","index_ver":1,"timestamp":"2021-03-12T02:54:01.180024320Z","ver":{"pool":9,"epoch":10},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-69:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.20"}]},{"op_id":"10#00000000001.44.5","op_tag":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6165.4505","op":"write","object":"b","instance":"","state":"complete","index_ver":1,"timestamp":"2021-03-12T02:54:01.131134003Z","ver":{"pool":9,"epoch":1},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e:bbucnh-69:6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.20"}]}] 2021-03-12T02:54:30.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:54:30.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-12T02:54:30.264 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-12T02:54:30.342 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:30.342 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:31.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:31.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'bbucnh-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:31.850 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket bbucnh-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:31.928 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:31.928 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:32.163 INFO:teuthology.orchestra.run.smithi077.stdout:[] 2021-03-12T02:54:32.164 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... ok 2021-03-12T02:54:34.034 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... ok 2021-03-12T02:54:34.035 INFO:tasks.rgw_multisite_tests: 2021-03-12T02:54:34.036 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-03-12T02:54:34.036 INFO:tasks.rgw_multisite_tests:Ran 28 tests in 1400.313s 2021-03-12T02:54:34.037 INFO:tasks.rgw_multisite_tests: 2021-03-12T02:54:34.037 INFO:tasks.rgw_multisite_tests:OK 2021-03-12T02:54:34.041 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2021-03-12T02:54:34.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:34.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:34.043 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:34.083 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:34.083 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:34.284 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:54:34.315 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517652.719188_1347.1","last_update":"2021-03-12T02:54:12.719188Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:54:34.315 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:54:34.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:34.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:34.317 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:34.357 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:34.358 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:34.567 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517652.719188_1347.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:12.719188Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:34.568 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:54:34.568 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:54:34.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:34.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:34.570 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:34.609 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:34.610 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:34.849 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517652.719188_1347.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:12.719188Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:34.850 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:54:34.851 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:54:34.851 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:34.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:34.852 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:34.893 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:34.893 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:35.159 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517652.719188_1347.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:12.719188Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:35.159 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:54:35.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:35.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:35.161 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:35.202 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:35.203 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:35.410 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:54:35.444 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517652.719188_1347.1","last_update":"2021-03-12T02:54:12.719188Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:54:35.444 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:54:35.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:35.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:35.447 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:35.491 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:35.492 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:35.754 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517652.719188_1347.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:12.719188Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:35.755 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:54:36.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:36.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:36.564 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:36.599 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:36.600 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:36.792 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:54:36.817 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517676.556706_1390.1","last_update":"2021-03-12T02:54:36.556706Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:54:36.818 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:54:36.819 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:36.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:36.820 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:36.863 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:36.864 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:37.130 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517676.556706_1390.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:36.556706Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:37.131 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:54:37.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:37.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:37.352 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:37.392 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:37.393 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:37.607 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:54:37.636 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517676.556706_1390.1","last_update":"2021-03-12T02:54:36.556706Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:54:37.637 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:54:37.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:37.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:37.639 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:37.679 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:37.679 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:37.939 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517676.556706_1390.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:36.556706Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:37.940 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:54:38.168 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... ok 2021-03-12T02:54:38.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:38.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:38.170 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:38.207 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:38.207 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:38.414 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:54:38.446 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517678.160426_1398.1","last_update":"2021-03-12T02:54:38.160426Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:54:38.446 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:54:38.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:38.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:38.448 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:38.489 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:38.489 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:38.687 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517678.160426_1398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:38.160426Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:38.688 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:54:38.688 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:54:38.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:38.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:38.690 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:38.729 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:38.730 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:38.972 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517678.160426_1398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:38.160426Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:38.973 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:54:38.973 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:54:38.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:38.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:38.975 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:39.017 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:39.017 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:39.284 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517678.160426_1398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:38.160426Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:39.285 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:54:39.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:39.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:39.287 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:39.329 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:39.329 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:39.536 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:54:39.566 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517678.160426_1398.1","last_update":"2021-03-12T02:54:38.160426Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:54:39.567 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:54:39.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:39.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:39.568 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:39.606 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:39.606 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:39.869 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517678.160426_1398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:38.160426Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:39.916 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:54:40.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:40.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:40.260 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:40.294 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:40.294 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:40.503 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:54:40.534 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517680.253124_1408.1","last_update":"2021-03-12T02:54:40.253124Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:54:40.535 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:54:40.535 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:40.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:40.536 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:40.577 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:40.578 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:40.928 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517680.240458_1399.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:40.240458Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:40.929 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1615517680.253124_1408.1 target=1_1615517680.240458_1399.1 2021-03-12T02:54:45.930 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:45.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:45.931 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:45.974 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:45.975 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:46.245 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517680.253124_1408.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:40.253124Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:46.245 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:54:46.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:46.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:46.287 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:46.323 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:46.324 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:46.547 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:54:46.578 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517680.253124_1408.1","last_update":"2021-03-12T02:54:40.253124Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:54:46.579 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:54:46.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:46.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:46.580 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:46.621 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:46.621 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:46.880 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517680.253124_1408.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:40.253124Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:46.880 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:54:47.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:47.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'bbucnh-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:47.804 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket bbucnh-72 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:47.843 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:47.844 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:48.094 INFO:teuthology.orchestra.run.smithi077.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.100.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.91.5"},{"key":4,"val":"00000000001.105.5"},{"key":5,"val":"00000000001.51.5"},{"key":6,"val":""},{"key":7,"val":"00000000001.65.5"},{"key":8,"val":"00000000001.142.5"},{"key":9,"val":"00000000001.95.5"},{"key":10,"val":"00000000002.203.5"}]} 2021-03-12T02:54:48.094 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-72 2021-03-12T02:54:48.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-72', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:48.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-72', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:48.096 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-72 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:48.141 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:48.141 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:48.402 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-12T02:54:48.403 WARNING:rgw_multi.tests:bucket bbucnh-72 zone test-zone4 behind zone test-zone1: shard=1 master=00000000002.100.5 target=, shard=3 master=00000000001.91.5 target=, shard=4 master=00000000001.105.5 target=, shard=5 master=00000000001.51.5 target=, shard=7 master=00000000001.65.5 target=, shard=8 master=00000000001.142.5 target=, shard=9 master=00000000001.95.5 target=, shard=10 master=00000000002.203.5 target= 2021-03-12T02:54:53.405 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-72', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:53.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'bbucnh-72', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:53.406 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket bbucnh-72 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:53.449 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:53.450 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:53.713 INFO:teuthology.orchestra.run.smithi114.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.100.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.91.5"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.105.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.51.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.65.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.142.5"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.95.5"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.203.5"}}}] 2021-03-12T02:54:53.713 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=bbucnh-72 2021-03-12T02:54:53.722 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-03-12T02:54:47.447085Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'bbucnh-72_notif', 'bucket': {'name': 'bbucnh-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::bbucnh-72', 'id': '6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.23'}, 'object': {'key': '2', 'size': 0, 'etag': '865e64e3', 'versionId': '', 'sequencer': 'F8D74A602A3E2121', 'metadata': [], 'tags': []}}, 'eventId': '1615517688.555826.865e64e3', 'opaqueData': ''} 2021-03-12T02:54:53.723 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-03-12T02:54:47.491177Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'bbucnh-72_notif', 'bucket': {'name': 'bbucnh-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::bbucnh-72', 'id': '6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.23'}, 'object': {'key': '3', 'size': 0, 'etag': '21ed294f', 'versionId': '', 'sequencer': 'F8D74A60A4D22721', 'metadata': [], 'tags': []}}, 'eventId': '1615517688.556257.21ed294f', 'opaqueData': ''} 2021-03-12T02:54:53.723 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-03-12T02:54:47.401865Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'bbucnh-72_notif', 'bucket': {'name': 'bbucnh-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::bbucnh-72', 'id': '6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.23'}, 'object': {'key': '1', 'size': 0, 'etag': 'b33331d1', 'versionId': '', 'sequencer': 'F8D74A60DC382A21', 'metadata': [], 'tags': []}}, 'eventId': '1615517688.556415.b33331d1', 'opaqueData': ''} 2021-03-12T02:54:53.723 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-03-12T02:54:47.535155Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'bbucnh-72_notif', 'bucket': {'name': 'bbucnh-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::bbucnh-72', 'id': '6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.23'}, 'object': {'key': '4', 'size': 0, 'etag': '03287a6f', 'versionId': '', 'sequencer': 'F8D74A6032199423', 'metadata': [], 'tags': []}}, 'eventId': '1615517688.596908.03287a6f', 'opaqueData': ''} 2021-03-12T02:54:53.724 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-03-12T02:54:47.715039Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'bbucnh-72_notif', 'bucket': {'name': 'bbucnh-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::bbucnh-72', 'id': '6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.23'}, 'object': {'key': '8', 'size': 0, 'etag': '16660db9', 'versionId': '', 'sequencer': 'F8D74A601A109723', 'metadata': [], 'tags': []}}, 'eventId': '1615517688.597102.16660db9', 'opaqueData': ''} 2021-03-12T02:54:53.724 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-03-12T02:54:47.379250Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'bbucnh-72_notif', 'bucket': {'name': 'bbucnh-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::bbucnh-72', 'id': '6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.23'}, 'object': {'key': '0', 'size': 0, 'etag': 'e4c3e237', 'versionId': '', 'sequencer': 'F8D74A60DFF10F26', 'metadata': [], 'tags': []}}, 'eventId': '1615517688.638579.e4c3e237', 'opaqueData': ''} 2021-03-12T02:54:53.725 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-03-12T02:54:47.670915Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'bbucnh-72_notif', 'bucket': {'name': 'bbucnh-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::bbucnh-72', 'id': '6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.23'}, 'object': {'key': '7', 'size': 0, 'etag': '25350b85', 'versionId': '', 'sequencer': 'F9D74A6026C25F1E', 'metadata': [], 'tags': []}}, 'eventId': '1615517689.509592.25350b85', 'opaqueData': ''} 2021-03-12T02:54:53.725 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-03-12T02:54:47.582618Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'bbucnh-72_notif', 'bucket': {'name': 'bbucnh-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::bbucnh-72', 'id': '6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.23'}, 'object': {'key': '5', 'size': 0, 'etag': '28e91973', 'versionId': '', 'sequencer': 'F9D74A602E137123', 'metadata': [], 'tags': []}}, 'eventId': '1615517689.594613.28e91973', 'opaqueData': ''} 2021-03-12T02:54:53.725 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-03-12T02:54:47.758654Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'bbucnh-72_notif', 'bucket': {'name': 'bbucnh-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::bbucnh-72', 'id': '6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.23'}, 'object': {'key': '9', 'size': 0, 'etag': '89e512f3', 'versionId': '', 'sequencer': 'F9D74A608C417223', 'metadata': [], 'tags': []}}, 'eventId': '1615517689.594690.89e512f3', 'opaqueData': ''} 2021-03-12T02:54:53.726 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2021-03-12T02:54:47.626859Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'bbucnh-72_notif', 'bucket': {'name': 'bbucnh-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::bbucnh-72', 'id': '6ca985fc-9a52-4c32-b5c5-4e09d94d7e1e.6169.23'}, 'object': {'key': '6', 'size': 0, 'etag': '5e686f83', 'versionId': '', 'sequencer': 'F9D74A60BDF74B26', 'metadata': [], 'tags': []}}, 'eventId': '1615517689.642512.5e686f83', 'opaqueData': ''} 2021-03-12T02:54:54.066 WARNING:tasks.rgw_multi.tests_ps:superfluous records are found 2021-03-12T02:54:54.227 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... ok 2021-03-12T02:54:54.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:54.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:54.229 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:54.272 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:54.273 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:54.475 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:54:54.510 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517694.218558_1419.1","last_update":"2021-03-12T02:54:54.218558Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:54:54.510 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:54:54.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:54.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:54.512 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:54.554 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:54.555 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:54:54.773 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517680.253124_1408.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:40.253124Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:54:54.774 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615517694.218558_1419.1 target=1_1615517680.253124_1408.1 2021-03-12T02:54:59.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:59.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:54:59.777 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:54:59.818 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:54:59.819 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:00.027 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517694.218558_1419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:54.218558Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:00.028 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:55:00.028 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:55:00.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:00.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:00.030 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:00.070 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:00.071 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:00.305 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517694.218558_1419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:54.218558Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:00.306 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:55:00.307 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:00.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:00.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:00.308 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:00.349 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:00.350 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:00.611 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517694.218558_1419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:54.218558Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:00.611 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:00.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:00.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:00.613 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:00.652 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:00.653 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:00.867 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:00.900 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517694.218558_1419.1","last_update":"2021-03-12T02:54:54.218558Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:00.900 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:00.901 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:00.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:00.902 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:00.946 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:00.946 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:01.205 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517694.218558_1419.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:54:54.218558Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:01.205 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:02.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:02.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:02.213 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:02.248 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:02.248 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:02.437 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:02.465 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517702.204751_1429.1","last_update":"2021-03-12T02:55:02.204751Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:02.466 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:02.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:02.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:02.468 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:02.507 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:02.507 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:02.762 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517702.204751_1429.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:02.204751Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:02.763 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:02.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:02.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:02.836 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:02.878 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:02.878 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:03.088 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:03.142 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517702.204751_1429.1","last_update":"2021-03-12T02:55:02.204751Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:03.143 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:03.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:03.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:03.144 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:03.186 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:03.186 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:03.454 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517702.204751_1429.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:02.204751Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:03.455 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:04.499 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... ok 2021-03-12T02:55:04.505 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:04.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:04.507 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:04.546 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:04.546 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:04.757 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:04.788 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517704.491227_1437.1","last_update":"2021-03-12T02:55:04.491227Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:04.789 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:55:04.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:04.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:04.790 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:04.835 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:04.836 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:05.041 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517702.204751_1429.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:02.204751Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:05.042 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615517704.491227_1437.1 target=1_1615517702.204751_1429.1 2021-03-12T02:55:10.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:10.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:10.044 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:10.087 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:10.088 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:10.285 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517704.491227_1437.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:04.491227Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:10.285 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:55:10.286 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:55:10.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:10.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:10.287 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:10.328 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:10.328 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:10.580 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517704.491227_1437.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:04.491227Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:10.580 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:55:10.581 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:10.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:10.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:10.582 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:10.623 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:10.624 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:10.882 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517704.491227_1437.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:04.491227Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:10.893 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:10.895 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:10.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:10.898 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:10.960 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:10.960 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:11.184 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:11.220 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517704.491227_1437.1","last_update":"2021-03-12T02:55:04.491227Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:11.220 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:11.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:11.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:11.222 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:11.266 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:11.267 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:11.530 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517704.491227_1437.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:04.491227Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:11.531 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:11.653 INFO:tasks.rgw_multisite_tests:test s3 topics set/get/delete on master ... ok 2021-03-12T02:55:11.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:11.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:11.656 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:11.695 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:11.696 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:11.900 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:11.935 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517704.491227_1437.1","last_update":"2021-03-12T02:55:04.491227Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:11.936 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:55:11.937 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:11.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:11.938 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:11.978 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:11.978 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:12.195 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517704.491227_1437.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:04.491227Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:12.195 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:55:12.196 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:55:12.196 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:12.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:12.198 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:12.238 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:12.238 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:12.496 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517704.491227_1437.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:04.491227Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:12.497 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:55:12.498 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:12.498 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:12.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:12.499 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:12.541 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:12.541 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:12.810 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517704.491227_1437.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:04.491227Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:12.810 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:12.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:12.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:12.812 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:12.850 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:12.851 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:13.063 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:13.095 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517704.491227_1437.1","last_update":"2021-03-12T02:55:04.491227Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:13.095 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:13.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:13.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:13.097 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:13.138 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:13.139 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:13.402 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517704.491227_1437.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:04.491227Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:13.403 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:13.403 INFO:tasks.rgw_multisite_tests:test s3 topics with secret set/get/delete on master ... ok 2021-03-12T02:55:13.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:13.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:13.405 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:13.447 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:13.448 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:13.657 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:13.689 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517704.491227_1437.1","last_update":"2021-03-12T02:55:04.491227Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:13.689 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:55:13.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:13.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:13.691 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:13.739 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:13.739 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:13.952 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517704.491227_1437.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:04.491227Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:13.953 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:55:13.953 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:55:13.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:13.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:13.955 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:13.990 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:13.991 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:14.221 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517704.491227_1437.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:04.491227Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:14.221 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:55:14.222 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:14.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:14.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:14.224 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:14.266 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:14.266 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:14.525 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517704.491227_1437.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:04.491227Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:14.526 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:14.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:14.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:14.528 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:14.565 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:14.566 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:14.782 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:14.813 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517704.491227_1437.1","last_update":"2021-03-12T02:55:04.491227Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:14.814 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:14.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:14.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:14.816 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:14.857 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:14.858 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:15.129 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517704.491227_1437.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:04.491227Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:15.129 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:16.186 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2021-03-12T02:55:16.186 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter_on_master ... ok 2021-03-12T02:55:16.187 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_filter ... ok 2021-03-12T02:55:16.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:16.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:16.189 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:16.226 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:16.227 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:16.411 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:16.442 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517716.177275_1461.1","last_update":"2021-03-12T02:55:16.177275Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:16.443 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:55:16.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:16.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:16.445 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:16.486 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:16.487 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:16.709 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517716.177275_1461.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:16.177275Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:16.709 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:55:16.710 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:55:16.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:16.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:16.712 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:16.754 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:16.755 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:17.016 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517716.177275_1461.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:16.177275Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:17.017 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:55:17.017 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:17.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:17.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:17.019 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:17.060 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:17.061 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:17.323 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517716.177275_1461.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:16.177275Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:17.324 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:17.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:17.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:17.325 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:17.365 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:17.365 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:17.589 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:17.624 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517716.177275_1461.1","last_update":"2021-03-12T02:55:16.177275Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:17.625 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:17.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:17.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:17.627 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:17.669 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:17.669 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:17.934 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517716.177275_1461.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:16.177275Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:17.934 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:18.280 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2021-03-12T02:55:18.281 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_objcet_timing ... ok 2021-03-12T02:55:18.281 INFO:tasks.rgw_multisite_tests:test pushing amqp s3 notification on master ... ok 2021-03-12T02:55:18.282 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2021-03-12T02:55:18.282 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2021-03-12T02:55:18.283 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl ... ok 2021-03-12T02:55:18.283 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl_sasl ... ok 2021-03-12T02:55:18.284 INFO:tasks.rgw_multisite_tests:test deletion of multiple keys on master ... ok 2021-03-12T02:55:18.284 INFO:tasks.rgw_multisite_tests:test pushing http s3 notification on master ... ok 2021-03-12T02:55:18.285 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2021-03-12T02:55:18.285 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification on master ... ok 2021-03-12T02:55:18.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:18.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:18.288 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:18.325 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:18.325 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:18.543 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:18.577 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517718.270780_1473.1","last_update":"2021-03-12T02:55:18.270780Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:18.577 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:55:18.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:18.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:18.579 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:18.623 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:18.623 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:18.843 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517718.270780_1473.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:18.270780Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:18.843 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:55:18.844 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:55:18.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:18.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:18.845 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:18.887 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:18.888 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:19.127 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517718.270780_1473.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:18.270780Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:19.127 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:55:19.128 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:19.128 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:19.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:19.129 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:19.170 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:19.170 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:19.439 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517718.270780_1473.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:18.270780Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:19.439 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:19.440 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:19.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:19.441 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:19.482 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:19.482 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:19.720 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:19.754 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517718.270780_1473.1","last_update":"2021-03-12T02:55:18.270780Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:19.754 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:19.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:19.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:19.756 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:19.795 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:19.796 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:20.061 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517718.270780_1473.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:18.270780Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:20.062 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:20.082 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... ok 2021-03-12T02:55:20.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:20.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:20.084 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:20.125 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:20.125 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:20.330 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:20.362 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517718.270780_1473.1","last_update":"2021-03-12T02:55:18.270780Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:20.362 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:55:20.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:20.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:20.364 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:20.404 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:20.405 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:20.623 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517718.270780_1473.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:18.270780Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:20.623 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:55:20.624 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:55:20.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:20.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:20.625 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:20.669 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:20.670 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:20.917 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517718.270780_1473.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:18.270780Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:20.917 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:55:20.918 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:20.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:20.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:20.920 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:20.961 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:20.962 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:21.235 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517718.270780_1473.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:18.270780Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:21.236 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:21.237 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:21.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:21.238 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:21.297 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:21.298 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:21.527 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:21.560 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517718.270780_1473.1","last_update":"2021-03-12T02:55:18.270780Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:21.561 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:21.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:21.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:21.563 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:21.609 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:21.873 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517718.270780_1473.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:18.270780Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:21.874 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:21.891 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... ok 2021-03-12T02:55:21.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:21.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:21.893 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:21.934 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:21.934 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:22.155 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:22.188 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517718.270780_1473.1","last_update":"2021-03-12T02:55:18.270780Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:22.188 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:55:22.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:22.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:22.190 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:22.230 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:22.231 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:22.444 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517718.270780_1473.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:18.270780Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:22.444 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:55:22.445 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:55:22.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:22.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:22.446 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:22.492 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:22.493 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:22.742 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517718.270780_1473.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:18.270780Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:22.742 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:55:22.743 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:22.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:22.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:22.744 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:22.788 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:22.789 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:23.048 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517718.270780_1473.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:18.270780Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:23.048 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:23.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:23.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:23.050 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:23.089 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:23.090 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:23.301 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:23.326 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517718.270780_1473.1","last_update":"2021-03-12T02:55:18.270780Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:23.327 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:23.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:23.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:23.328 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:23.370 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:23.370 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:23.636 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517718.270780_1473.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:18.270780Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:23.637 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:24.638 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:24.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:24.639 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:24.675 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:24.675 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:24.884 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:24.915 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517724.630461_1495.1","last_update":"2021-03-12T02:55:24.630461Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:24.916 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:24.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:24.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:24.917 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:24.958 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:24.958 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:25.219 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517718.270780_1473.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:18.270780Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:25.220 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=2 master=1_1615517724.630461_1495.1 target=1_1615517718.270780_1473.1 2021-03-12T02:55:30.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:30.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:30.223 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:30.271 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:30.272 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:30.527 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517724.630461_1495.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:24.630461Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:30.528 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-12T02:55:30.942 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... ok 2021-03-12T02:55:30.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:30.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:30.945 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:30.983 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:30.983 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:31.187 INFO:teuthology.orchestra.run.smithi077.stderr:No --period given, using current period=8d2c0f1d-7035-4313-b22b-345f1f4f81db 2021-03-12T02:55:31.224 INFO:teuthology.orchestra.run.smithi077.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1615517730.934007_1500.1","last_update":"2021-03-12T02:55:30.934007Z"},{"marker":"","last_update":"0.000000"}] 2021-03-12T02:55:31.225 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-12T02:55:31.225 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:31.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:31.226 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:31.269 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:31.269 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:31.479 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517724.630461_1495.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:24.630461Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:31.479 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1615517730.934007_1500.1 target=1_1615517724.630461_1495.1 2021-03-12T02:55:36.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:36.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:36.482 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:36.526 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:36.527 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:36.725 INFO:teuthology.orchestra.run.smithi114.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517730.934007_1500.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:30.934007Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:36.726 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-12T02:55:36.726 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-12T02:55:36.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:36.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:36.728 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:36.769 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:36.769 INFO:teuthology.orchestra.run.smithi077.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T02:55:37.010 INFO:teuthology.orchestra.run.smithi077.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8d2c0f1d-7035-4313-b22b-345f1f4f81db","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1615516156.395861_98.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-12T02:29:16.395861Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1615517730.934007_1500.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:55:30.934007Z","realm_epoch":4}},{"key":3,"val":{"state":1,"marker":"1_1615516309.597935_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-12T02:31:49.597935Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-12T02:55:37.011 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-12T02:55:37.011 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-12T02:55:37.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:37.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-12T02:55:37.013 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-12T02:55:37.053 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setuser ceph since I am not root 2021-03-12T02:55:37.054 INFO:teuthology.orchestra.run.smithi114.stderr:ignoring --setgroup ceph since I am not root 2021-03-12T14:21:53.514 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-03-12T14:21:53.582 DEBUG:teuthology.task.console_log:Killing console logger for smithi077 2021-03-12T14:21:53.584 DEBUG:teuthology.task.console_log:Killing console logger for smithi114 2021-03-12T14:21:53.584 DEBUG:teuthology.exit:Finished running handlers